PaperlayerView exact revision on arXiv

Source-linked AI summary

MemFit: Efficient Long-Term Agentic Memory

Mitchell Piehl, Muchao Ye

arXiv:2610.00872v1cs.AI

TL;DR

Existing memory systems often use frequent LLM calls to organize and consolidate memory, creating costly write operations, while semantic-only retrieval can admit irrelevant matches. MemFit stores turns verbatim and combines LLM-free hybrid retrieval with reranking; it reports strong benchmark performance with reduced construction cost and time.

  • Problem

    Memory systems that rely on frequent LLM calls for consolidation and storage incur costly write operations, and semantic-similarity retrieval can bring irrelevant memories into results.

    Source: §1 · p. 2

  • Method

    MemFit stores each turn verbatim in an append-only episode store, uses segment summaries as indexes, and retrieves with lexical and semantic signals, candidate expansion, and cross-encoder reranking.

    Source: §1 · p. 2, §3.2 · pp. 4–5

  • Results

    MemFit reports a superior balance of retrieval efficiency and reasoning correctness on LoCoMo, MemGallery, and LongMemEval-S across evaluated backbones, while outperforming more expensive alternatives and generalizing across architectures and scales.

    Source: §1 · p. 2, §5 · pp. 9–10

  • Takeaways & Limitations

    The reported results indicate that MemFit is a promising solution for real-time agentic systems.

    Source: §5 · p. 10

    Without storage-time LLM consolidation or organization, retrieving and combining memories across sessions is harder and performance is slightly decreased on questions requiring those abilities; the current system handles only text and images.

    Source: E LIMITATIONS · p. 20

Abstract

from arXiv · show

Long-term memory systems for large language models (LLMs) have gained popularity for extending reasoning capabilities across applications. Current memory systems rely on LLM agents to organize and consolidate memory, resulting in costly, inefficient write operations. To address this limitation, we propose MemFit, a long-term memory system for conversational agents that reduces the cost and latency of memory operations. Unlike existing systems that rely on expensive LLM calls for memory construction or discard surface-level details through compression, MemFit stores each turn verbatim in an append-only store with near-instantaneous, LLM-free insertion, indexing turns with segment summaries rather than replacing them. Additionally, MemFit uses an LLM-free, multi-path retrieval strategy that combines lexical and semantic signals with cross-encoder reranking over caption- augmented episodes in both textual and multimodal settings. Empirical results on three widely used benchmarks, LoCoMo, MemGallery, and LongMemEval-S, show that MemFit achieves state-of-the-art performance while reducing memory construction time and cost several-fold, providing a scalable and efficient solution for persistent agentic memory.

1 INTRODUCTION

LLMs’ fixed context windows make reasoning over long prompts difficult, while memory systems that organize or consolidate memories with frequent LLM calls raise efficiency and retrieval-precision concerns. MemFit keeps turns verbatim and uses LLM-free storage and retrieval to pursue efficient storage and precise retrieval.

1 INTRODUCTION

  • Frequent LLM calls for memory consolidation and storage, alongside retrieval based predominantly on semantic similarity, raise cost and context-pollution concerns.Context pollution occurs when irrelevant but semantically similar memories are retrieved.Source: §1 · p. 2
  • MemFit stores each turn verbatim in an append-only episode store, indexed lexically and semantically, while segment summaries index rather than replace the stored turns.It uses one summary call per conversational segment; insertion otherwise requires an encoder pass rather than an LLM call.Source: §1 · p. 2
  • MemFit combines hybrid retrieval, additive candidate-pool expansions, and cross-encoder reranking without LLM calls for searching.Caption-augmented episodes also support retrieval of visual content through the same lexical and semantic pathways.Source: §1 · p. 2
  • MemFit reports a superior balance of retrieval efficiency and reasoning correctness on LoCoMo, MemGallery, and LongMemEval-S across the evaluated backbones.The reported configuration spans three model families.Source: §1 · p. 2

2 RELATED WORK

Prior work develops persistent memory through graph, incremental, hierarchical, and consolidation-based designs, while other approaches target efficiency or iterative retrieval. MemFit instead preserves verbatim turns, uses summaries as indexes, and performs retrieval without LLM guidance.

Memory systems for LLMs

  • Persistent memory approaches include A-MEM’s linked graph and Mem0’s incremental extraction and consolidation, while SimpleMem and LightMem target efficiency through compression and sleep-time maintenance, respectively.MemFit contrasts with these approaches by keeping turns verbatim without per-turn LLM calls and using segment summaries to index them.Source: §2 · p. 2
  • Multimodal retrieval is established in RAG but remains underexplored in agentic memory; MemFit embeds detailed visual descriptions in episode text for unified text-based search.The paper contrasts this with standard cross-modal embedding approaches.Source: §2 · p. 2

Memory summarization and consolidation

  • Hierarchical summarization systems organize conversations at multiple levels, while consolidation systems may alter stored content through forgetting or deletion.MemFit segments conversations without an LLM and keeps summaries as indexes rather than retrieving compressed segments as memory units.Source: §2 · p. 3

Iterative retrieval for LLMs

  • Iterative retrieval methods use sequential LLM feedback, whereas MemFit combines normalized lexical and dense scores, pseudo-relevance feedback, and cross-encoder reranking without LLM guidance.The cited iterative methods include Self-RAG, FLARE, and IRCoT.Source: §2 · p. 3

3 METHODOLOGY

MemFit stores timestamped conversational turns as persistent episodes and retrieves relevant evidence for answer generation. Segment summaries index, rather than replace, verbatim turns; retrieval combines lexical and semantic search, additive expansions, and cross-encoder reranking.

3.1 PROBLEM STATEMENT

  • MemFit represents dialogue history as timestamped utterances and retrieves a small subset of relevant episodes to ground answers to factual, temporal, multi-hop, or multimodal queries.The system must locate episodes, reconcile updates across sessions, and connect visual content to conversational context.Source: §3.1 · p. 4

3.2 MEMORY STORAGE AND THE SUMMARY LAYER

  • Each utterance becomes a timestamped, speaker-labeled episode in an append-only store with BM25 and dense indexes for exact-term and paraphrase matching.Inserting an episode adds its postings and embedding without rewriting, re-embedding, or re-summarizing stored episodes; sequence numbers preserve temporal order.Source: §3.2 · pp. 4–5
  • TextTiling segments consecutive episodes, and each segment of at least three episodes receives a summary whose provenance links it to the verbatim turns it describes.The summary is generated in one LLM call and indexes the turns rather than compressing or replacing them.Source: §3.2 · p. 5
  • MemFit keeps changed facts as separate dated episodes instead of merging, rewriting, or deleting them, deferring which statement matters until retrieval.Retrieved competing statements are ordered chronologically, while summaries describe stretches of conversation without standing in for their turns.Source: §3.2 · p. 5

3.3 MEMORY RETRIEVAL FOR ANSWER EXTRACTION

  • Retrieval scores episodes through lexical and semantic paths, expands candidates using summary provenance and pseudo-relevance feedback, then reranks candidates with a cross-encoder.Answer extraction uses chronologically ordered excerpts with dates and resolved relative times; multimodal answering passes ranked images in the same reader call.Source: §3.3 · pp. 5–6
  • Insertion uses an embedding and two index updates, while the only write-time LLM cost is one call per segment; retrieval’s sparse scan and matrix-vector product grow linearly with memory size.Neither insertion per turn nor retrieval per search step requires an LLM call.Source: §3.3 · p. 6

4 EXPERIMENTS AND RESULTS

Across three benchmarks, MemFit is reported to outperform leading memory systems, while experiments also assess its efficiency and retrieval components. Results include a trade-off between lower memory-building costs and longer reads of verbatim turns.

4.1 EXPERIMENTAL SETTINGS

  • Evaluation covers LoCoMo, MemGallery, and LongMemEval-S, with multiple proprietary, open-source, and multimodal backbones and benchmark-specific metrics.One configuration is used across backbones and benchmarks without per-corpus tuning; implementation uses lexical and semantic candidate paths, cross-encoder reranking, and a k=10 episode pack with a one-episode context window.Source: §4.1 · pp. 6–7

4.2 COMPARISON RESULTS

  • 44.19% vs. 33.45% average F1 with Qwen3-8B; MemFit leads most LoCoMo categories across backbones, but trails SimpleMem on GPT-4.1-mini multi-hop (41.58% vs. 43.46%).On LoCoMo, the same average-F1 ordering is reported for GPT-4.1-mini (50.26% vs. 43.24%), GPT-4o (49.33% vs. 39.06%), and Gemma3-27B (45.42% vs. 28.98%).Source: Table 1 · p. 7, §4.2 · pp. 7–8

4.3 COMPUTATIONAL EFFICIENCY ANALYSIS

  • 3.9–52.6× faster memory building (946s vs. 3,708–49,765s); MemFit is faster than every baseline to build memory, but its 1.23s per-question latency is not the fastest.Against systems consulting an LLM as memories arrive, it issues 81–87% fewer LLM calls and costs 82–88% less; versus LightMem, write-time costs are 5.7× lower, while full-run costs are about the same ($1.53 vs. $1.46).Source: Figure 3 · p. 8, §4.3 · p. 9

4.2 COMPARISON RESULTS

  • MemFit achieves the best overall performance on all three benchmarks and with every backbone; on LongMemEval-S, it correctly answers 64.60% of 500 questions, beating every baseline.The reported LongMemEval-S margins are 0.31 points over LightMem, 2.0 over A-MEM, 3.6 over naive retrieval, 7.8 over full-history reading, and 11.0 over Mem0.Source: §4.2 · pp. 7–8, Table 2 · p. 8

4.4 ABLATION STUDY

  • Removing the context window causes the largest reported ablation drop: 1.86 average-F1 points and 1.7 judge-score points; removing all components costs 3.32 F1 and 5.5 judge points.Pseudo-relevance feedback, summary expansion, the wide first-stage scan, and summaries each account for 0.83–1.35 average-F1 points; the components overlap in recovered evidence.Source: §4.4 · p. 9

4.5 SENSITIVITY TO RETRIEVAL SETTINGS

  • Across retrieval settings, judge accuracy stays between 60.1% and 62.3%; k=5 lowers average F1 by 1.35 points, while k=20 or 30 adds no gain and can make prompts 2.65× longer.The paper therefore uses k=10.Source: §4.5 · p. 9, Table 3 · p. 9

5 CONCLUDING REMARKS

MemFit combines efficient storage with retrieval designed to improve performance while reducing memory-construction costs. Its pseudocode specifies incremental episode storage, off-query segment summaries, and a staged retrieval process that uses an LLM only to answer the query.

5 CONCLUDING REMARKS

  • MemFit unifies efficient storage and effective retrieval, with experiments reporting strong performance and lower computational cost during memory insertion.The conclusion describes consistent generalization across model architectures and scales and identifies real-time agentic systems as a promising application.Source: §5 · pp. 9–10
  • Because MemFit stores conversations verbatim and never deletes them automatically, deployment could retain sensitive information or injected content, requiring safeguards.The paper reports using public benchmarks and no new human subjects or personal data.Source: ETHICS STATEMENT · p. 10

A PSEUDOCODE

  • Each utterance becomes a timestamped, speaker-labeled episode; optional image identifiers and captions are added before encoding and indexing, without an LLM call.Episodes are appended to the store, BM25 index, and embedding matrix; stored episodes are not rewritten.Source: A PSEUDOCODE · p. 13
  • Off the query path, MemFit segments episodes using embedding-similarity minima and summarizes qualifying segments with one LLM call each.The pseudocode specifies segments of at least three episodes, summaries from up to 14 evenly spaced excerpts, and a limit of 60 summaries per conversation.Source: A PSEUDOCODE · p. 13
  • Retrieval combines lexical and semantic candidates, expands candidates using summaries and rare query terms, reranks them, then uses one LLM call to answer.The final context includes top-ranked episodes and their nearby session neighbors, with excerpts arranged chronologically and dates resolved.Source: A PSEUDOCODE · p. 13

B.1 DATASETS

The benchmarks cover long conversational histories, factual and temporal reasoning, and multimodal questions, with benchmark-specific evaluation protocols. The comparison includes memory systems with different storage and retrieval designs, as well as full-context and plain-retrieval baselines.

B.1 DATASETS

  • All three benchmarks use the same storage procedure: one episode per turn, followed by segment summaries, without benchmark-specific tuning.The accompanying table is identified as a summary of benchmark statistics.Source: §B.1 · p. 13, Table 4 · p. 14
  • LoCoMo contains 10 conversations spanning 19–32 sessions and 5,882 turns; the evaluation uses 1,540 of its 1,986 questions, excluding adversarial questions.The reader receives date-resolved excerpts and questions, and the reported comparison uses specified backbone and judge configurations.Source: §B.1 · pp. 14–15
  • MemGallery includes 20 multimodal scenarios, 240 sessions, 3,962 rounds, and 1,711 questions; its rounds pair dialogue with image identifiers and captions.Evaluation reports token-level F1 and LLM-judge accuracy across all 1,711 questions.Source: §B.1 · p. 14
  • LongMemEval-S contains 500 questions over timestamped histories averaging 47.7 sessions and roughly 115k tokens, including 30 abstention questions.Its six question types include single- and multi-session recall, preferences, knowledge updates, and temporal reasoning.Source: §B.1 · pp. 14–15

B.2 COMPARATIVE BASELINES

  • Baselines span full-context reading, plain chunk retrieval, and memory systems that use LLM calls for extraction, consolidation, or organization.The comparison table identifies where each baseline is evaluated; MemGallery and LongMemEval-S baseline results come from prior reports, while Figure 3 efficiency measurements are collected by the authors.Source: Table 5 · p. 15, §B.2 · pp. 15–16

C ADDITIONAL RESULTS

The appendix reports per-category results across configurations, including results omitted from the main text. It describes the default run protocol, judge scoring, and paired testing used to compare configurations.

C ADDITIONAL RESULTS

  • The appendix reports per-category results for every configuration and includes results not shown in the main text.Unless stated otherwise, runs use the main-text configuration and make one LLM call per question.Source: C ADDITIONAL RESULTS · p. 16
  • LoCoMo and MemGallery LLM-judge accuracy is scored by gpt-oss-20b using the prompt in Appendix F.Source: C ADDITIONAL RESULTS · p. 16
  • Configuration differences are tested with paired sign-flip permutation tests over per-question judge verdicts, with gained/lost counts tracking discordant answers.Source: C ADDITIONAL RESULTS · p. 16

C.1 LOCOMO

On LoCoMo, MemFit’s memory design improves judge accuracy and average F1 over plain retrieval across five readers. Ablations and retrieval analyses indicate where components help, while the sensitivity results describe the cost of changing pack size.

C.1 LOCOMO

  • 4.6 to 7.9 judge points and 2.8 to 4.5 average F1: MemFit’s memory design outperforms plain retrieval across five readers.The comparison uses the same store, encoders, cross-encoder, and reader; differences in judge points are reported with p < 0.001.Source: §C.1 · p. 16, Table 6 · p. 17
  • Open-domain F1 is 18.42 with all components versus 15.14 to 17.09 without any one; multi-hop loses 1.0 to 1.6 F1 without a component and 3.8 with plain retrieval.The context window accounts for most of the single-hop gain, while temporal scores barely move; open-domain per-component differences are noisy because the category has 96 questions.Source: §C.1 · p. 17, Table 7 · p. 17, Table 8 · p. 17
  • 82.5% versus 79.2%: MemFit increases the share of questions whose retrieved pack contains gold evidence; top-30 hit rate rises from 83.4% to 88.5%.Pack-hit gains are largest for multi-hop questions (74.8% to 81.2%) and open-domain questions (51.1% to 57.6%).Source: §C.1 · pp. 17–18, Table 9 · p. 18
  • k=5 costs 3.12 judge points (p < 0.001), while k=20 and k=30 change judge score by +0.85 (p=0.37) and −0.13 (p=0.95), at 1.85× and 2.65× input tokens.The sensitivity study varies one setting at a time; Table 10 reports per-category and average metrics, judge differences, and input tokens relative to the default.Source: §C.1 · p. 18, Table 10 · p. 18

C.2 MEMGALLERY AND LONGMEMEVAL-S

MemFit’s MemGallery results show a benefit from including the question’s image, while adding retrieved-episode images brings no further overall gain. The ablation is largely flat on MemGallery, and LongMemEval-S memory construction uses substantially fewer LLM calls than several baselines.

C.2 MEMGALLERY AND LONGMEMEVAL-S

  • Adding retrieved-episode images provides no further overall MemGallery gain (67.47), while shifting category results across Test-Time Learning, Knowledge Resolution, and Visual-centric Search.The question-image condition reaches 67.66 F1 overall, compared with 67.47 when retrieved-episode images are also passed.Source: §C.2 · p. 18
  • Using one category-blind format instruction instead of per-category instructions lowers MemGallery F1 from 64.82 to 47.92; the reported Table 11 results use the official protocol.Table 11 reports token-level F1 by category under that protocol.Source: §C.2 · p. 18, Table 11 · p. 19
  • 2.85 F1 is added by passing the question’s image to the reader over text alone, rising from 64.82 to 67.66; judge accuracy also rises by 2.63 (p < 0.001).Gains concentrate in Test-Time Learning (72.7 to 82.4) and Visual-centric Reasoning (45.2 to 52.7).Source: §C.2 · p. 18, Table 11 · p. 19
  • MemGallery’s ablation is flat: no component significantly changes judge accuracy, and plain retrieval places gold evidence in the pack for 95.5% of questions.With Qwen2.5-VL-7B, plain retrieval is within 1.52 judge points of the full system (p=0.094); with Qwen3-8B, its gain is +3.27 judge points (p=0.001).Source: §C.2 · p. 18, Table 12 · p. 19
  • 40 LLM calls per LongMemEval-S history are 13–73× fewer than LangMem, Mem0, A-MEM, and MemoryOS, with fewer tokens than LightMem.Table 13 reports memory-construction cost per history, excluding the answer call.Source: Table 13 · p. 19, §C.2 · p. 19

C.3 COST OF EVERY STAGE

The cost breakdown separates fast turn storage and segmentation from summary calls, while reporting retrieval and answer-time costs. The broader-impact discussion describes potential benefits alongside risks from persistent verbatim storage and unverified content.

C.3 COST OF EVERY STAGE

  • 15 s for all LoCoMo conversations and 18 s for all MemGallery scenarios cover turn storage and indexing; segmenting them takes under 0.2 s.The remaining memory-build time comes from summary calls.Source: §C.3 · pp. 19–20
  • 87 to 112 ms per question is the retrieval time, with about 60–65% spent in the cross-encoder; answer calls take the rest.MemFit’s answer prompt is 2.07–2.10× as long as plain retrieval’s on LoCoMo and 1.64× on MemGallery.Source: §C.3 · p. 20, Table 14 · p. 20

D BROADER IMPACTS

  • Long-term memory could support accessibility, management of complex personal information, educational research, and long-horizon collaboration without current architectures’ prohibitive costs.These are stated as potential benefits.Source: D BROADER IMPACTS · p. 20
  • Verbatim append-only storage preserves sensitive details and unverified or malicious text, which may later reach the reader as retrieved evidence; deployment needs deletion, retention limits, and access control.User-initiated deletion requires removing an episode from both indexes and re-summarizing its segment.Source: D BROADER IMPACTS · p. 20

E LIMITATIONS

The stated limitations are harder cross-session retrieval and support limited to text and images. The prompt templates specify segment summarization, answer extraction, and the evaluation judge’s instructions.

E LIMITATIONS

  • Without LLM consolidation or organization during storage, retrieving and combining memories across sessions is harder at retrieval time and slightly lowers performance on questions requiring those abilities.The current system handles text and image modalities only.Source: E LIMITATIONS · p. 20

F PROMPT TEMPLATES

  • Storage uses one LLM call per conversational segment to summarize what the stretch is about and what is currently true, using up to 14 evenly spaced excerpts.The prompt requests two sentences and names people, places, and things explicitly; each excerpt is truncated to 220 characters.Source: F PROMPT TEMPLATES · p. 21
  • Retrieval uses one answer-extraction call per question; it supplies chronological excerpts and image guidance, and asks the model to reason from excerpts rather than decline when inference is needed.The instructions also specify when to return NOT_FOUND and how to format concise answers.Source: F PROMPT TEMPLATES · p. 21
  • In multimodal mode, the prompt distinguishes an image attached to the question from images in conversation excerpts and identifies their order.On MemGallery, the benchmark system prompt is prepended, with category-specific answer-format instructions for three categories.Source: F PROMPT TEMPLATES · p. 21
  • The evaluation judge compares the student answer with the question and correct answer, marking omissions, contradictions, or declines as wrong and returning JSON.It allows different wording and extra true context when the answer conveys the same information.Source: F PROMPT TEMPLATES · p. 22

G EXPERIMENT COMPUTE RESOURCES

Experiments ran on a single server with four NVIDIA RTX PRO 6000 Blackwell GPUs, with open-weight models served through Ollama and some models accessed through the OpenAI API. The paper reports per-run timings and total experiment compute.

G EXPERIMENT COMPUTE RESOURCES

  • 946 s builds all ten LoCoMo conversations with Gemma3-27B on one GPU; each question takes 1.95 s with eight model instances sharing four GPUs.A full LoCoMo configuration took about 8.5 minutes, while MemGallery with Qwen2.5-VL-7B took about 5 minutes text-only or 20 minutes with images.Source: G EXPERIMENT COMPUTE RESOURCES · p. 22
  • 43 s is the average memory-build time per LongMemEval-S history with Qwen3-8B, and answering takes 1.1 s per question.A LoCoMo run on GPT-4.1-mini or GPT-4o sends 2.94M input tokens and receives 10–14k output tokens.Source: G EXPERIMENT COMPUTE RESOURCES · p. 22
  • 10.5 hours of wall-clock time, or roughly 42 GPU-hours, covered the reported experiments, including ablations, sensitivity studies, readers, cost measurements, and judging.Open-weight models ran on one server with four NVIDIA RTX PRO 6000 Blackwell GPUs; GPT-4.1-mini, GPT-4o, and GPT-4o-mini used the OpenAI API.Source: G EXPERIMENT COMPUTE RESOURCES · p. 22

Related papers

Loading 2610.00872v1…