Source-linked AI summary
LycheeMemory V2: Efficient Long-Term Memory for LLM Agents via Semantic Segment-Level Consolidation
Dongfang Li, Zixuan Liu, Junmai Wang, Jiahe Huang, Fuhao Li, Bonian Jia, Baotian Hu, Min Zhang
TL;DR
Long-term LLM-agent memory must preserve fine-grained evidence without making construction and retrieval increasingly expensive. LycheeMemory V2 uses semantic segment-level consolidation and achieves stronger accuracy–cost trade-offs than representative baselines across LoCoMo and LongMemEval-S.
Problem
Long-term agent memory must preserve fine-grained evidence while controlling both construction and retrieval costs.
Method
LycheeMemory V2 batches exchanges into semantically bounded segments, encoding each into context-independent typed records for structured, query-planned retrieval.
Results
Across LoCoMo and LongMemEval-S, LycheeMemory improves long-term memory QA while delivering a stronger accuracy–cost trade-off than representative baselines.
Takeaways & Limitations
Long-term memory performance depends on the granularity of evidence consolidation, not only on the information retained.
Takeaways & Limitations
LycheeMemory remains weaker on preference-intensive questions than specialized memory baselines.
Abstract
from arXiv · showhide
Long-horizon LLM agents must preserve information from past interactions to support future tasks. Existing memory systems typically rely on eager consolidation, invoking LLMs after each interaction to extract, summarize, or update memories. This design makes memory construction increasingly costly as conversations grow. Coarse summarization can reduce construction cost but risks discarding fine-grained contextual evidence, whereas larger retrieval contexts or multi-hop LLM reasoning shift the overhead to query time. We present LycheeMemory V2, an efficient long-term memory framework that replaces turn-level consolidation with semantic segment-level consolidation. Instead of consolidating every interaction, LycheeMemory batches multiple exchanges into segments and encodes each finalized segment into context-independent typed memory records. Segment-level batching lowers LLM encoding frequency, while semantic boundary detection helps preserve coherent event-level and temporal evidence compared with fixed-window batching. The resulting records are organized with lightweight structured indexes for query-planned evidence retrieval. Experiments using GPT-4.1-Mini show that LycheeMemory achieves state-of-the-art performance, reaching 89.22% on LoCoMo and 92.20% on LongMemEval-S. Compared with A-Mem, it reduces construction tokens by 86.0% on LoCoMo and 75.9% on LongMemEval-S without increasing query-time token usage. More broadly, our results suggest that the accuracy--cost trade-off of long-term agent memory depends not only on what information is retained, but also on the granularity at which it is consolidated.
1. Introduction
Long-horizon LLM agents need external memory because active context windows cannot reliably retain unbounded interaction histories. LycheeMemory V2 addresses the cost–evidence trade-off by consolidating semantically coherent segments into typed records rather than eagerly processing every turn.
- Eager turn-level consolidation repeatedly invokes language models, increasing memory-construction cost as conversations grow.
- LycheeMemory V2 batches exchanges into semantically coherent segments and encodes each finalized segment once into typed evidence records for structured retrieval.
- Persistent agents must remember preferences, events, evolving facts, constraints, and prior failures across long-horizon interactions.
- External memory modules compensate for bounded active context by extracting, updating, organizing, and retrieving information from past conversations.
- Coarse summaries can discard fine-grained entities, temporal expressions, coreference relations, and contextual details, while broader retrieval or multi-hop search shifts overhead to query time.
2. Related Work
Related work differentiates long-term memory systems by construction, organization and retrieval, and cost control. LycheeMemory V2 targets write-side efficiency through semantic segment-level construction while preserving coherent, self-contained evidence.
- Efficiency and results: LycheeMemory achieves over 20-point accuracy gains over A-Mem with up to 7.2× fewer construction tokens and lower query-time token usage.The reported evaluation covers LoCoMo and LongMemEval-S, with particularly large gains on multi-hop, temporal-reasoning, and preference-related questions.
- Research axes: Long-term memory research is organized around memory construction, organization and retrieval, and cost control during long-horizon use.These three technical axes structure the related-work discussion.
- Memory construction: Eager systems construct memories from incoming dialogue, enriching write-time semantics but requiring frequent LLM-mediated ingestion or summarization.Mem0 and A-Mem operate at turn granularity, while related hierarchical systems retain LLM-mediated ingestion or summarization.
- Memory construction: Cost-oriented methods reduce write-side expense through post-construction compression, deterministic management, or learned routing, trading semantic flexibility or pipeline characteristics.MemRefine uses budgeted deletion, merging, or preservation; DMF removes LLM calls from management; MemRouter learns embedding-based routing.
- Memory construction: Segment-level systems such as SeCom and HiMem improve construction granularity, while LycheeMemory uses semantic boundaries to share encoding calls across multiple exchanges.LycheeMemory combines segment-level batching with bounded cross-segment disambiguation to construct self-contained records online.
- Memory organization and retrieval: Retrieval research adds structured stores, hierarchical representations, and query-time operations to support temporal, multi-hop, and cross-episode reasoning beyond flat semantic retrieval.Examples include graph-like organization, multi-granularity workspaces, iterative evidence retrieval, query routing, and reasoning-aware reranking.
3. Method
LYCHEEMEMORY frames long-term agent memory as online construction and retrieval, replacing turn-level consolidation with semantic segment-level encoding. It combines embedding-based segmentation, context-independent typed records, structured evidence indexes, and query planning to control construction and query-time costs.
- Construction overview: The system buffers exchanges into coherent segments and runs one encoding pass only when each segment is finalized.Semantic boundaries follow conversational structure rather than fixed windows, reducing encoding frequency while preserving event-level coherence.
- Semantic segmentation: A segment is finalized when its boundary score exceeds threshold δ or reaches a hard token cap; otherwise, the new exchange remains buffered.The score combines semantic surprise, cohesion drop, token pressure, and turn-count pressure, using embeddings without mid-segment LLM inference.
- Segment encoding: Each finalized segment is encoded into self-contained typed records that extract atomic information, resolve references, and normalize relative times.Records include entities, topics, temporal information, provenance, and types such as facts, preferences, events, constraints, procedures, failure patterns, and tool affordances.
- Segment encoding: A bounded recent-record reference context preserves cross-segment continuity without allowing prompt size to grow with the full conversation history.The encoder carries resolved aliases, canonical entity names, and reference relations into later segments while truncating context to a fixed budget.
- Evidence organization and retrieval: Structured indexes complement vector retrieval by supporting evidence access through entity, topic, time, event context, and combinations of these fields.At query time, only the planning call is generative; recall, expansion, reranking, fusion, and selection use lookups, filtering, or arithmetic scoring.
4 Experiments
Across LoCoMo and LongMemEval-S, LycheeMemory achieves the highest overall accuracy with both backbones. With GPT-4.1-Mini, it reaches 89.22% and 92.20% while reducing construction and query-time token consumption.
- Overall performance: 89.22% on LoCoMo and 92.20% on LongMemEval-S are LycheeMemory’s GPT-4.1-Mini overall accuracies, exceeding the strongest respective baselines.The strongest baselines are Full Context at 84.80% on LoCoMo and TiMem at 75.80% on LongMemEval-S.
- LoCoMo: 27.3 pp and 25.0 pp are LycheeMemory’s largest GPT-4.1-Mini gains over A-Mem on LoCoMo multi-hop and open-domain questions.It also reaches 93.34% on single-hop questions, 20.1 pp above A-Mem.
- LongMemEval-S: 34.59 pp, 26.67 pp, and 26.32 pp are LycheeMemory’s largest GPT-4.1-Mini gains over A-Mem on LongMemEval-S temporal, preference-tracking, and multi-session reasoning questions.Knowledge-update accuracy reaches 97.44%, compared with 82.05% for A-Mem.
- Efficiency: 204.1K construction tokens on LoCoMo and 304.7K on LongMemEval-S are 86.0% and 75.9% below A-Mem, respectively.Segment-level batching reduces encoding calls from one per turn to one per segment.
- Efficiency: 4.01K and 8.88K are LycheeMemory’s query-token totals on LoCoMo and LongMemEval-S, below A-Mem by 27.9% and 42.6%, respectively.The retrieval pipeline requires only one LLM planning call and does not expand query-time context or add multi-step LLM reasoning.
- Ablation: 89.22% falls to 81.88% under eager construction, while construction tokens rise from 204.1K to 849.9K; fixed-window consolidation reaches 82.40%.Fixed-window consolidation uses 174.7K construction tokens and loses most on multi-hop and open-domain questions.
5. Conclusion
LycheeMemory replaces turn-level consolidation with semantic segment-level consolidation, encoding coherent segments into context-independent typed records for efficient long-term memory. Ablations show that its construction, representation, retrieval, and boundary components jointly support accuracy–cost gains, while performance remains stable across evaluated thresholds.
- Memory representation: Summary-level records reduce construction tokens to 99.7K but lower accuracy to 80.78% (-8.4 pp) and temporal reasoning to 73.83% (-12.8 pp).Removing cross-segment reference context leaves construction tokens nearly unchanged at 189.6K vs. 204.1K but reduces accuracy to 81.56% (-7.7 pp).
- Retrieval pipeline: Record-vector-only retrieval keeps query tokens comparable at 4.09K vs. 4.01K but reduces accuracy to 81.75% (-7.5 pp).Removing the query planner reduces query tokens to 2.26K but lowers accuracy to 83.38% (-5.8 pp), exposing a planning–accuracy trade-off.
- Boundary-threshold sensitivity: 1.04 percentage points: accuracy ranges from 88.18% to 89.22% as the boundary threshold varies from 0.30 to 0.70.The default threshold δ=0.50 achieves the highest accuracy, indicating stability across the evaluated range.
- Summary: Removing or replacing proposed construction, representation, and retrieval components causes substantial accuracy losses or cost increases, whereas threshold variation preserves stable accuracy.The conclusion attributes the gains to the overall system design rather than narrowly tuned boundary thresholds.
- System contribution: LycheeMemory replaces turn-level consolidation with semantic segment-level batching, semantic boundary detection, and context-independent typed records.The design also includes lightweight cross-segment disambiguation, structured evidence indexes, and query-planned multi-route recall.
A Experimental Protocol Details
The appendix defines the evaluation protocol for Section 4, covering benchmark composition, answer generation, judge configuration, and token accounting. Unless stated otherwise, generation and judging use deterministic decoding with temperature 0.
- Evaluation Protocol: The protocol specifies benchmark composition, answer generation, judge configuration, and token accounting rules.These details support interpretation of the reported accuracy–cost trade-off without hidden implementation assumptions.
- Evaluation Protocol: Unless otherwise stated, all generation and judge calls use deterministic decoding with temperature set to 0.
A.1 Benchmark Data and Evaluation Tasks · A.1.1 LoCoMo
This section distinguishes LoCoMo from LongMemEval-S by evaluation unit and describes LoCoMo as a dense, multi-session conversational-memory benchmark. The main LoCoMo evaluation uses 1,540 standard semantic-answering question-answer pairs drawn from the first four categories.
- A.1 Benchmark Data and Evaluation Tasks: LoCoMo evaluates multiple question-answer pairs over each multi-session conversation, unlike LongMemEval-S, which evaluates one question per long conversation-question instance.The benchmarks use different evaluation units and question taxonomies, so their statistics are reported separately.
- A.1 Benchmark Data and Evaluation Tasks: Aggregating LoCoMo and LongMemEval-S statistics into one table would obscure their different sources of difficulty.The separation reflects differences in both evaluation structure and question taxonomy.
- A.1.1 LoCoMo: LoCoMo is a long-context conversational-memory benchmark built from companion-style multi-session dialogues.Each dialogue contains two named speakers.
- A.1.1 LoCoMo: LoCoMo tests recovery of fine-grained personal facts, speaker-specific reference resolution, and evidence combination across turns or sessions.These requirements make the benchmark target distributed conversational memory evidence.
- A.1.1 LoCoMo: 10 multi-session conversations comprise LoCoMo, averaging approximately 600 turns and 16K tokens per conversation.The benchmark is small in conversation count but dense in annotated memory questions.
- A.1.1 LoCoMo: 1,986 question-answer pairs are provided in total, while the main evaluation retains 1,540 after excluding adversarial/counterfactual questions.The retained set comes from the standard long-term memory categories.
- A.1.1 LoCoMo: The main LoCoMo evaluation uses 1,540 valid question-answer pairs from the first four categories.Category 5 adversarial/counterfactual questions are excluded because they are outside the standard semantic-answering setting.
- A.1.1 LoCoMo: The same 1,540 questions support the main results, cost analysis, and LoCoMo ablations, enabling identical-set comparisons across method variants.Table 6 provides the category-level composition.
A.1.2 LongMemEval-S · A.2 Answer Generation and Judge Protocol
LongMemEval-S evaluates memory systems on 500 long conversation-question instances averaging approximately 115K tokens, covering diverse factual, preference, cross-session, update, and temporal-reasoning demands. Answer generation uses GPT-4.1-Mini and GPT-4o-Mini under a shared benchmark split and judge framework, with task-specific evaluation rules for LongMemEval-S.
- A.1.2 LongMemEval-S: LongMemEval-S tests recovery of user facts, assistant-side facts, preferences, cross-session evidence, changed information, and temporal relations.These demands arise from agent-style, task-oriented interactions with substantially longer dialogue histories.
- A.1.2 LongMemEval-S: LongMemEval-S contains 500 conversation-question instances, each pairing one question with a long dialogue history.Unlike LoCoMo, it is organized at the instance level rather than as multiple QA pairs over the same conversation.
- A.1.2 LongMemEval-S: Approximately 115K tokens per instance makes LongMemEval-S substantially longer than LoCoMo and suitable for testing memory construction under high context load.The benchmark’s longer histories increase the context burden faced by memory systems.
- A.1.2 LongMemEval-S: The evaluation uses all 500 questions, reports category accuracy by official question type, and computes the overall score as a micro-average.Table 7 abbreviates the categories as SSU, SSA, SSP, MS, KU, and TR.
- A.2 Answer Generation and Judge Protocol: GPT-4.1-Mini and GPT-4o-Mini generate answers for all methods using the same benchmark split and judge protocol.Each system first constructs or retrieves an evidence context according to its design; LycheeMemory also uses the selected model for memory encoding and query planning.
- A.2 Answer Generation and Judge Protocol: For LoCoMo, the judge receives the question, gold answer, and generated answer, accepts semantically equivalent phrasings, and computes the overall score over 1,540 retained questions.Category accuracy is computed within each retained question type.
- A.2 Answer Generation and Judge Protocol: LongMemEval-S uses its official task-specific yes/no judge protocol with a 10-token output cap for judge responses.Standard single-session and multi-session questions use answer containment, preference questions use a rubric-style desired response, and knowledge-update questions accept updated answers that also mention previous information.
- A.2 Answer Generation and Judge Protocol: Temperature is set to 0 for all generative components, and token accounting includes generative LLM input and output tokens reported in thousands (K).These rules are specified alongside the evaluation components and metric definitions.
A.3 Metrics and Token Accounting … B.2 Plan-Guided Multi-Route Retrieval
The appendix defines evaluation and token-accounting rules, then specifies how LycheeMemory constructs segment-level memories and performs plan-guided multi-route retrieval. These procedures distinguish generative LLM calls from non-generative retrieval operations and describe how evidence context is assembled.
- A.3 Metrics and Token Accounting: Accuracy is computed over 1,540 retained LoCoMo questions and 500 LongMemEval-S questions, with no filtering beyond each benchmark’s valid evaluation set.The reported metrics include category accuracy, overall accuracy, construction tokens, and query tokens.
- A.3 Metrics and Token Accounting: Construction tokens cover memory-building calls, while query tokens cover answer generation and LycheeMemory’s single query-planning call.All reported token costs use GPT-4.1-Mini.
- B Algorithmic Description: The appendix records when generative LLM calls occur, which intermediate states are maintained, and how the final evidence context is assembled.It provides algorithm-level descriptions of LycheeMemory’s two core procedures.
- Algorithm B.1: Semantic Segment-Level Memory Construction: Algorithm B.1 initializes an active segment, same-session reference context, and local surprise history before processing the conversation stream.Its inputs include the conversation stream, session identifier, memory store, and boundary threshold.
- Algorithm B.1: Semantic Segment-Level Memory Construction: Finalizing a segment encodes it with reference context, normalizes records, inserts them into memory, builds metadata-based evidence nodes, and updates same-session context.The normalized record fields include type, text, entities, tags, temporal information, and provenance.
- B.1 Semantic Segment-Level Memory Construction: Segment-level batching assigns one encoding call per finalized segment, while semantic boundary detection uses embeddings and deterministic scoring to determine segment composition.Cross-segment disambiguation is carried forward only as reference context, not as a new source of facts.
- B.2 Plan-Guided Multi-Route Retrieval: Algorithm B.2 uses one LLM planning call to convert a question into typed evidence routes; route recall, filtering, search, reranking, fusion, and selection are non-generative operations.The non-generative stages include temporal filtering, record search, raw-turn search, reciprocal-rank fusion, and diversity-aware selection.
C Implementation Details … D.2 Query Tokens
The implementation uses semantic segment batching, typed context-independent memory records, structured evidence indexes, and plan-guided multi-route retrieval. Token accounting distinguishes write-side construction cost from query-time generative context cost.
- C Implementation Details: The four-stage pipeline comprises online semantic segmentation, segment-level memory encoding, structured evidence organization, and plan-guided multi-route retrieval.These stages follow the implementation order described in the appendix.
- C.1 Online Semantic Segmentation: Semantic batching shares encoding calls across multiple exchanges, finalizing segments on saturation, topic transitions, or hard capacity limits.Segmentation uses text-embedding-3-small and maintains centroid, recent-embedding, semantic-surprise, length, and exchange-count state.
- Algorithm B.2: Plan-Guided Multi-Route Retrieval: Retrieval generates one structured plan, selects a question-type strategy, applies temporal filters when specified, and gathers evidence across planned routes.The effective evidence budget is k←max(k, k_plan, 1), with route-specific query variants and candidate sets.
- C.2 Segment-Level Memory Encoding: Finalized segments are encoded into typed records using the segment text plus compact same-session reference context, producing updated disambiguation state.The records are designed for retrieval and interpretation without the original dialogue context.
- C.3 Structured Evidence Organization: Structured organization builds entity, topic, entity-topic, temporal, and event-frame indexes through embeddings, SQLite/FTS indexing, and deterministic bookkeeping without a generative LLM.Textually distinct, successive, or conflicting statements remain separate, while conflict resolution is deferred to retrieval and answer generation.
- C.4 Plan-Guided Multi-Route Retrieval: Routes combine direct-record, evidence-node, temporal, and raw-turn recall, then use reciprocal-rank fusion, route-coverage quotas, and diversity-aware selection.The final evidence budget is count-based top-k rather than a fixed token budget, so query prompt length varies with selected evidence text.
- D Token Accounting Details: Construction tokens measure all recorded generative LLM input and output tokens used during memory building, averaged per conversation or conversation-question instance in thousands (K).For LYCHEEMEMORY, this includes memory-encoding prompts, segment text, reference context, records, and disambiguation-state outputs, excluding non-generative operations.
- D.2 Query Tokens: Query tokens measure recorded generative LLM input and output tokens used to answer evaluation questions, averaged per question in thousands (K).They include planning, answer generation, serialized evidence, and recent dialogue context, while excluding retrieval, filtering, reranking, selection, and judge calls.
E Ablation Variant Definitions … E.3 Retrieval-Side Variants
The appendix defines ablation variants by removing or replacing construction, representation, and retrieval modules while preserving the remaining system components where specified. These variants isolate the effects of batching, semantic boundaries, typed records, cross-segment context, recall routes, query planning, and evidence selection.
- E Ablation Variant Definitions: The appendix specifies which modules each Section 4.4 ablation removes, replaces, or preserves.The main text reports the results, while the appendix defines the implementations.
- E.1 Construction-Side Variants: Per-turn construction replaces segment-level batching with turn-level eager construction while keeping the record schema, structured indexing, and query-time retrieval unchanged.It uses speaker-turn granularity on LoCoMo and original message/exchange granularity on LongMemEval-S.
- E.1 Construction-Side Variants: Fixed-window consolidation replaces semantic boundary detection with mechanical batching using 600 target chunk tokens and at most 10 exchanges.Batching is preserved to compare semantic boundaries with fixed windows at a comparable scale.
- E.2 Representation-Side Variants: Summary-level records replace typed, self-contained records with summaries that omit explicit memory type, entity, topic, temporal scope, and provenance fields.The unchanged retrieval pipeline enables comparison between typed records and summary-level memory.
- E.3 Retrieval-Side Variants: Retrieval-side variants include record-vector retrieval only, without a query planner, and without fusion, reranking, or diversity-aware selection.These variants respectively isolate structured and raw-turn recall, adaptive query planning, and the combined evidence-selection stack.
- E.2 Representation-Side Variants: Without cross-segment reference context, each segment is encoded independently without same-session disambiguation context or recent resolved records.The record schema and retrieval pipeline remain unchanged.
- E.3 Retrieval-Side Variants: Record-vector retrieval only disables entity, topic, temporal, event-frame, and raw-turn multi-route recall, retaining only memory-record vector search.The final evidence budget remains consistent with the requested top-k of the full system.
- E.3 Retrieval-Side Variants: Without a query planner, the system uses a fixed single-route query with no LLM rewriting, question-type classification, or route decomposition.Query tokens decrease, but routes cannot adapt to temporal, comparison, personalized-advice, or prior-assistant-response intents.
F Prompt Templates and Evaluation Prompts
The appendix specifies LycheeMemory’s segment-encoding, query-planning, and answer-generation prompts, alongside the official evaluation judges for LongMemEval-S and LoCoMo. The prompts constrain evidence use, retrieval planning, temporal handling, conflict resolution, and evaluation parsing.
- F Prompt Templates and Evaluation Prompts: LycheeMemory’s prompt templates cover memory encoding, query planning, answer generation, and the LoCoMo and LongMemEval-S evaluation judges.The templates are drawn from the implementation, while LongMemEval-S follows its official yes/no evaluator and LoCoMo uses its JSON label protocol.
- F.1 LYCHEEMEMORY Segment Encoding Prompt: Segment encoding extracts self-contained atomic memory records from current conversation turns, using reference context only to resolve references and aliases.The user prompt includes session date, reference context, and current turns; reference context is not a source for new facts.
- F.2 LYCHEEMEMORY Query Planning Prompt: Query planning converts the visible question and recent context into an executable JSON retrieval plan without deciding answer availability or inventing candidates.Ordinary named-speaker factual questions default to one route, while multiple routes are reserved for explicitly separate evidence needs.
- F.3 LYCHEEMEMORY Answer Generation Prompt: Answer generation receives episodic/semantic and raw memory blocks, avoids double-counting, focuses on the named person, and prioritizes recent supported information when memories conflict.The instructions also require attention to timestamps, direct evidence for factual questions, and person-specific signals for recommendations.
- F.4 LongMemEval-S Official Judge Prompts: LongMemEval-S judging uses GPT-4o-Mini at temperature 0 with a 10-token cap, parsing responses containing “yes” as correct.The evaluator constructs task-specific prompts from the task, question, answer, response, and abstention setting.
G Limitations and Artifact Use
The evaluation is limited to text-only conversational memory and does not assess several deployment concerns, while LYCHEEMEMORY remains weaker on preference-intensive questions. The artifact guidance also requires separate reporting of implementation components and compliance with benchmark licensing constraints.
- Evaluation scope: The evaluation excludes multimodal memories, production latency, online user feedback, privacy governance, cache behavior, and storage growth under continuous deployment.LYCHEEMEMORY primarily optimizes construction-token cost and query-time token overhead; deployment-oriented evaluation is still needed for database latency, embedding-index storage, and production monitoring.
- Preference-intensive questions: 90.00% on LongMemEval-S with GPT-4.1-Mini trails MemoryOS at 100.00%, while 70.00% with GPT-4o-Mini trails MemOS at 96.67%.These results identify a remaining weakness on preference-intensive questions and motivate stronger persona or user-profile modeling alongside segment-level evidence construction.
- Model and service assumptions: Hosted LLM and embedding APIs plus a reranker can be replaced with compatible open-source models, but accuracy and token accounting may change.Any replacement should be evaluated with newly reported accuracy and cost numbers.
- Artifact use: Artifact release should separate code, prompts, configurations, question-id lists, prediction files, and token-accounting summaries from benchmark redistribution.Artifacts containing LoCoMo source conversations, QA pairs, evidence, or substantial derived content should retain attribution and license notices and respect the CC BY-NC 4.0 non-commercial restriction.