PaperlayerView exact revision on arXiv

Source-linked AI summary

Recall What Is Missing: Composing Evidence from Long-Term Agent Memory

Qi Feng, Jicong Fan

arXiv:2610.15642v1cs.AI

TL;DR

Long-term agents must answer queries from limited contexts, yet pointwise memory selection can repeat relevant facts while leaving complementary evidence out. MemPursuit composes evidence through residual-guided query reconstruction using existing embeddings. It improves Top-10 accuracy by 7.2 percentage points in controlled LoCoMo comparisons and shows gains across benchmarks and memory frameworks.

  • Problem

    Pointwise memory selection can fill a limited context with repeated evidence while omitting a less similar fact needed to complete a multi-hop query.

    Source: §1 · p. 1, §2.2 · p. 3

  • Method

    MemPursuit uses selected memory embeddings to reconstruct the query and directs further recall with the residual, without additional language-model or cross-encoder scoring.

    Source: §1 · p. 2

  • Results

    7.2 percentage points: MemPursuit improves Top-10 accuracy in controlled LoCoMo comparisons and achieves the highest aggregate scores on LoCoMo, LongMemEval-S, and BEAM with both answerers.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Residual-guided composition is reported as an effective, efficient, and reusable operation for long-term agent memory, with gains across memory frameworks and benchmarks.

    Source: §1 · p. 2, §6 · p. 10

    MemPursuit relies on semantic embedding-space structure and cannot explicitly reason about contextual relationships between memories and the query.

    Source: §6 · p. 10

Abstract

from arXiv · show

Under a limited context budget, a long-term agent must recall not merely memories that are individually relevant, but evidence that is collectively sufficient for the current query. Conventional memory access retrieves, scores, and truncates memories, often spending multiple context slots on redundant facts while leaving complementary evidence behind. We cast the read phase of long-term memory as query-conditioned evidence composition: each memory is valued by what it adds beyond the evidence already selected. We instantiate this operation with MemPursuit, a training-free mechanism that uses the residual of a sparse convex query reconstruction as a representation-space proxy for what the current memory set is missing. By reusing the query and candidate embeddings already available from retrieval, MemPursuit progressively composes nested evidence contexts without invoking a language model or cross-encoder to score each query--memory pair. The effectiveness of our method is verified on 3 popular benchmarks in comparison to strong baselines. In fixed-budget LoCoMo experiments, MemPursuit improves end-to-end accuracy by 7.2 percentage points at Top-10 and 5.5 percentage points at Top-20, with consistent gains across multiple memory frameworks, while achieving a 32.9$\times$ speedup in median reranking latency over fully batched BGE-v2-m3. Our implementation is available at https://github.com/FengQi-HITSZ/MemPursuit.

1 INTRODUCTION

Long-term agents must assemble limited answer contexts from growing memory stores, but selecting individually relevant records can repeat evidence and omit complementary facts. MemPursuit selects memories according to what they add to the evidence already chosen, improving results across benchmarks and memory frameworks.

1 INTRODUCTION

  • Pointwise selection can spend multiple slots on repeated facts while omitting a less similar fact needed to complete a multi-hop evidence chain.Figure 1 contrasts repeated records about Deborah’s father and an Anna distractor with complementary records about Deborah’s father, mother, and friend Karlie.Source: §1 · p. 1, Figure 1 · p. 2
  • MemPursuit uses sparse convex query reconstruction: selected memory embeddings represent the query, and the residual guides complementary selections through iterative selection and refitting.The method reuses existing embeddings and needs no additional language-model or cross-encoder scoring.Source: §1 · p. 2
  • 7.2 percentage points: MemPursuit improves Top-10 accuracy in controlled LoCoMo comparisons and achieves the highest aggregate scores on LoCoMo, LongMemEval-S, and BEAM with both answerers.The controlled comparisons fix the memory store, candidate pool, and downstream evaluation.Source: §1 · p. 2
  • Complete gold-session coverage rises from 77.1% to 84.5% at Top-10, with higher answer accuracy using fewer memories.Reconstruction analysis connects the evidence gains to the geometric selection signal.Source: §1 · p. 3
  • Across three benchmarks and two answerers, accuracy gains occur in all eight framework–budget configurations in cross-framework comparisons.Reranker comparisons show competitive answer quality at a fraction of the selection cost.Source: §1 · p. 3

2 PRELIMINARIES

Memory writing determines what an agent retains, while memory reading chooses which stored records to use together for a query. This section frames reading as budgeted set selection and motivates a joint objective because individual relevance scores do not account for overlap or complementarity.

2.1 WRITING MEMORIES, READING EVIDENCE

  • Memory writing constructs and updates a persistent store from past interactions, whereas reading assembles stored information into the answer model’s context for a query.The writer determines what is retained; the reader chooses which memories to use together.Source: §2.1 · p. 3
  • For each query, the backbone supplies a candidate pool and the read policy selects at most K entries for the answer context; controlled comparisons hold the store, pool, budget, answerer, and evaluation fixed.The study focuses on this final selection step, not memory writing, organization, or candidate generation.Source: §2.1 · p. 3

2.2 MEMORY READING AS BUDGETED SET SELECTION

  • Pointwise Top-K selection gives each memory the same individual score regardless of other selected memories, so its additive objective misses overlap and complementary evidence.Because answer quality is unavailable at selection time, query reconstruction provides a computable joint signal for choosing a memory combination.Source: §2.2 · p. 3

3 MEMPURSUIT

MEMPURSUIT composes a limited memory context by reconstructing the query from selected embeddings and using the residual to guide further selection. It refits selected memories jointly, supports adaptive context sizes, and has a conditional evidence-coverage guarantee.

3.1 EVIDENCE COMPOSITION THROUGH SPARSE QUERY RECONSTRUCTION

  • MEMPURSUIT represents the query with a convex combination of selected memory embeddings, coupling each memory’s contribution to the rest of the set.Similar embeddings provide similar reconstruction directions, while complementary embeddings can bring the joint reconstruction closer to the query.Source: §3.1 · pp. 3–4

3.2 FULLY CORRECTIVE RESIDUAL PURSUIT

  • Fully corrective pursuit expands the selected set using residual alignment, then refits all selected weights so each next choice depends on the joint reconstruction.The first selection is the strongest query cosine match; the residual guides subsequent expansions.Source: §3.1 · p. 4, §3.2 · pp. 4–5

3.3 CONTEXT CONSTRUCTION AND ADAPTIVE STOPPING

  • A single pursuit path yields nested contexts for smaller budgets, while adaptive stopping can end selection when relative reconstruction improvement remains at or below tolerance τ.Selection stops at that threshold or when the memory budget is reached.Source: §3.3 · p. 5

3.4 FROM RECONSTRUCTION TO EVIDENCE COVERAGE

  • Under the stated fact-cluster separation and coverage-margin conditions, the first h selections cover all h required facts without distractors, so every prefix with K ≥ h covers them.The proposition assumes unit-norm embeddings and exact refits with fixed λ ≥ 0.Source: §3.4 · p. 5

3.5 COMPLEXITY ANALYSIS

  • MEMPURSUIT replaces repeated token-level neural evaluation with matrix operations on existing embeddings and small selected-set refits, removing dependence on scorer depth and width.The reported complexity comparison covers final memory selection, excluding query encoding, candidate retrieval, and answer generation.Source: §3.5 · pp. 5–6

4 EXPERIMENTS

The experiments evaluate MemPursuit across three benchmarks, two answerers, and multiple memory frameworks. They report strong end-to-end results, improved evidence coverage and accuracy under limited budgets, and competitive answer quality at lower reranking latency.

4.1 EXPERIMENTAL SETUP

  • The evaluation compares nine baselines across LoCoMo, LongMemEval, and BEAM, using two answerers and testing integration into four memory frameworks.The protocol uses a shared answer prompt, judge model, and judge prompt; full-system comparisons retain each baseline’s native memory construction and retrieval settings.Source: §4 · p. 6, §4.1 · p. 6

4.2 END-TO-END PERFORMANCE AND CROSS-FRAMEWORK COMPARISONS

  • MemPursuit with a Mem0-v3 backbone achieves the highest aggregate score on all three benchmarks with both answerers.With GPT-4o-mini, it reaches 87.08% on LoCoMo, 71.20% on LongMemEval-S, and 43.82 on BEAM.Source: §4.2 · p. 6, Table 2 · p. 7
  • At Top-10, MemPursuit improves accuracy in all eight framework–budget configurations, with gains ranging from 2.47% to 9.29%.The comparisons use paired systems with the same selection budget; improvements also hold at Top-20.Source: §4.2 · pp. 6–7, Table 3 · p. 7

4.3 DOES MEMPURSUIT RECALL WHAT IS MISSING?

  • Complete gold-session coverage rises from 77.1% to 84.5% at Top-10 and from 86.2% to 90.3% at Top-20.On LoCoMo, MemPursuit replaces 70.8% of selected memories at Top-10 and 64.5% at Top-20; newly recalled memories add more gold-session coverage than displaced memories lose.Source: §4.3 · p. 8, Figure 3 · p. 8

4.4 PERFORMANCE ACROSS QUESTION TYPES AND CONTEXT BUDGETS

  • Multi-hop accuracy rises from 63.1% to 75.9%, while single-hop accuracy increases from 79.7% to 87.9%.At a smaller context budget, MemPursuit reaches 85.1% accuracy with 20 memories, exceeding hybrid retrieval with 50.Source: §4.4 · pp. 8–9, Figure 4 · p. 8

4.5 QUALITY–EFFICIENCY TRADE-OFFS WITH NEURAL RERANKERS

  • MemPursuit captures approximately 73% of BGE-v2-m3’s accuracy gain over hybrid retrieval while requiring only 3.2% of its median reranking latency.It achieves a 32.9× median reranking-latency speedup over fully batched BGE-v2-m3.Source: §4.5 · p. 9, Figure 5 · p. 9

4.6 ADDITIONAL EXPERIMENTAL RESULTS

  • Additional analyses examine query reconstruction from gold evidence, residual scoring, case studies, MMR comparisons, and latency scaling with candidate count and budget.The passage locates these analyses in Appendix C.Source: §4.6 · p. 9

5 RELATED WORK

Related work covers memory management and organization, relevance ranking and set-dependent selection, and dynamic retrieval. MemPursuit focuses on composing the final context from a candidate pool through query reconstruction and residual-guided selection.

Long-term memory for language-model agents

  • Long-term agent-memory systems manage retention through persistent memory, segmentation and compression, linked or graph-structured organization, broader memory architectures, and reinforcement learning.MemPursuit instead targets composition of the final answer context from a supplied candidate pool, as a reusable read operation across architectures.Source: §5 · p. 9

From relevance ranking to evidence composition

  • Prior selection methods range from individual relevance and listwise ranking to set-dependent novelty, coverage, diversity, and direct evidence-utility approaches.MemPursuit evaluates the selected set through query reconstruction, with jointly refitted weights determining the residual that guides each addition.Source: §5 · p. 9

Dynamic retrieval and residual pursuit

  • Dynamic retrieval methods adapt to accumulated evidence, while MemPursuit performs this adaptation geometrically within the candidate pool.It alternates residual-guided selection with fully corrective simplex refitting, producing nested contexts through operations on existing embeddings.Source: §5 · p. 10

6 CONCLUSION

MemPursuit composes limited answer contexts through simplex-constrained residual pursuit, using selected memories to reconstruct the query and guide subsequent recall. Experiments report accuracy gains across memory frameworks and competitive answer quality at lower selection cost, with a limitation in representing contextual relationships explicitly.

6 CONCLUSION

  • MemPursuit uses simplex-constrained residual pursuit to compose complementary evidence into limited answer contexts.It reuses existing embeddings to produce nested contexts without additional neural scoring; its semantic embedding space does not explicitly reason about contextual relationships between memories and the query.Source: §6 · p. 10

A.1 MEMPURSUIT ALGORITHM AND IMPLEMENTATION

The fixed-budget procedure returns memories in entry order rather than sorting them by final coefficient magnitude.

A.1 MEMPURSUIT ALGORITHM AND IMPLEMENTATION

  • Fixed-budget MemPursuit returns selected memories in entry order, not by final coefficient magnitude.Source: Figure 6 · p. 14

A.1.1 NUMERICAL IMPLEMENTATION

This material specifies the pursuit implementation and its computational scope, experimental memory configurations and prompts, and assumptions used in the theoretical analysis.

A.1.1 NUMERICAL IMPLEMENTATION

  • Each refit warm-starts from the preceding solution, while the returned greedy path retains selected memories even if their fitted weights later become zero.The path is not reordered by final coefficient magnitude; adaptive stopping returns all entries selected before termination.Source: §A.1.1 · p. 14

A.2 COMPUTATIONAL COMPLEXITY

  • A single pursuit path supplies every smaller context budget, with total time O(P^2d + KP^2 + K^4) and working memory O(Pd + P^2).The cost bound covers selection from available candidates and embeddings, excluding query encoding, candidate retrieval, and answer generation.Source: §A.2 · pp. 14–15

A.3 MEMORY BACKBONES AND BASELINE CONFIGURATIONS

  • Mem0-v3 builds timestamped memories incrementally, retrieves a dense-and-BM25 candidate union, and supplies its top 200 memories to MEMPURSUIT.Selected memories are sorted chronologically for presentation; the comparison setups also preserve each framework’s specified native pool and processing, with paired pursuit variants using the same stored memories and candidate pools.Source: §A.3 · pp. 15–16
  • The system comparisons retain baseline-native retrieval settings, including different memory budgets and supplemental context for A-MEM, MemoryOS, and other frameworks.MemoryBank’s system-comparison budget is raised from 2 to 50; cross-framework K=10 and K=20 comparisons use prefixes of the same pursuit sequence.Source: §A.3 · p. 15

A.4.1 LOCOMO

  • LoCoMo answer prompts direct the model to scan and cross-reference memories, prioritize specific details, resolve temporal questions, and answer from available evidence.The answer-and-judge template comes from Mem0; its builder presents up to 200 memories oldest first, while the correctness judge receives the question, gold answer, and generated answer.Source: §A.4.1 · p. 16

A.4.2 LONGMEMEVAL

  • LongMemEval uses Mem0’s answer-generation and correctness-judging prompts, while BEAM uses prompts consistent with its official paper and implementation.The LoCoMo, LongMemEval-S, and BEAM prompt sources are specified separately.Source: §A.4.2 · p. 18, §A.4.3 · p. 18

B.2 FACT-CLUSTER MODEL

  • The fact-cluster model assumes a multi-fact query, alternative memories for each required fact, and separation conditions on similarities and distractors.One representative from each fact cluster suffices to make that fact available in context; selection itself uses embeddings and the fitted residual, not the model’s labels or parameters.Source: §B.2 · pp. 19–20

B.3 PROOF OF PROPOSITION 1

The proof relates residual-guided selection to evidence coverage under a fact-cluster model, then gives a computable certificate for the refitted objective.

B.3 PROOF OF PROPOSITION 1

  • Under the stated separation conditions, the first h selections represent all h required-fact clusters, and coverage persists in every longer returned prefix.The induction shows that a memory from an uncovered cluster outranks redundant candidates and distractors at each step.Source: §B.3 · p. 20

B.4 A COMPUTABLE OPTIMALITY CERTIFICATE

  • The certificate’s gap is zero exactly when the refitted coefficients minimize the regularized objective over the full candidate simplex.It bounds the remaining error in that objective and is computed from residual scores and fitted weights already used by MEMPURSUIT.Source: §B.4 · p. 21

C.1 FULL CROSS-FRAMEWORK RESULTS

  • Table 4 compares paired systems with and without MEMPURSUIT at K=10 and K=20 on LoCoMo, reporting accuracy and percentage-point changes across 1,540 questions.The caption describes the comparison layout but supplies no cell values.Source: Table 4 · p. 22, §C.1 · p. 22

C.2 RECONSTRUCTION RESIDUALS AND GOLD EVIDENCE

The reconstruction analysis compares gold evidence with equally sized hybrid retrieval sets and connects residual geometry to complementary evidence selection.

C.2 RECONSTRUCTION RESIDUALS AND GOLD EVIDENCE

  • Gold evidence has lower reconstruction residuals than hybrid Top-Kgold on many queries, with pronounced advantages on the positive side of the distribution.Figure 7 plots paired residual advantage: positive values indicate a lower residual for gold evidence, negative values for the baseline, with both sets evaluated using the same simplex refit.Source: §C.2 · p. 22, Figure 7 · p. 22
  • The residual comparison links evidence content to query fit and supports the mechanism in which refitting accounts for represented content while residuals guide recall toward uncovered content.Together with coverage gains, the analysis connects the selection geometry to complementary evidence in the answer context.Source: §C.2 · pp. 22–23

C.3 RESIDUAL PREFERENCE FOR MISSING EVIDENCE

This block examines residual-guided evidence composition through singleton comparisons and missing-fact cases, and compares MEMPURSUIT with MMR at fixed LoCoMo budgets.

C.3 RESIDUAL PREFERENCE FOR MISSING EVIDENCE

  • The residual test covers 136 LongMemEval questions and 381 singleton comparisons, with one mean utility difference per question in the histogram.The test retains questions with evidence in at least two answer sessions and missing evidence remaining after duplicate-fact filtering.Source: Figure 8 · p. 23, §C.3 · p. 23
  • At K=10, MEMPURSUIT selected the user's age at step 6 to complete 75 - 32; on LoCoMo, step 3 recovered Toby's adoption, enabling the three-month interval.Both answerers returned the correct age difference in the LongMemEval example.Source: §C.3 · p. 24, Table 5 · p. 24, Table 6 · p. 24
  • MEMPURSUIT promoted gastritis from hybrid rank 41 to step 6; on BEAM, it selected the omitted survey date at step 2, yielding the correct 18-day interval.The diagnosis example concerns recovery of the benchmark-annotated condition, not a medical causal relationship.Source: §C.3 · p. 25, Table 7 · p. 25, Table 8 · p. 25

C.4 QUESTION-TYPE RESULTS AT TOP-20

  • The Top-20 question-type table compares accuracy for hybrid ordering, MEMPURSUIT, and pointwise neural rerankers.The supplied caption gives the table's scope but not its cell values.Source: Table 9 · p. 26

C.5 COMPARISON WITH MAXIMAL MARGINAL RELEVANCE

  • At Top-10, MEMPURSUIT accuracy rises from 77.08% with MMR to 81.04%, a 3.96-point gain; the Top-20 gain is 1.75 points.MEMPURSUIT has higher accuracy at all three tested budgets.Source: §C.5 · p. 26, Table 10 · p. 26

C.6 ROBUSTNESS TO EMBEDDING CHOICE

The embedding-choice comparison finds MEMPURSUIT applicable across tested representation families, with stronger accuracy from Qwen3-Embedding-8B at tighter budgets.

C.6 ROBUSTNESS TO EMBEDDING CHOICE

  • All tested encoders outperform the fixed hybrid baseline; Qwen3-Embedding-8B further improves accuracy at Top-10 and Top-20.E5 and BGE remain close to the default encoder; candidate memories are held fixed while the embedding encoder changes.Source: §C.6 · p. 26, Table 11 · p. 27

C.7 READ-TIME EFFICIENCY

The efficiency results report MEMPURSUIT's reranking latency against neural scorers and measure its latency across candidate and selection budgets.

C.7 READ-TIME EFFICIENCY

  • MEMPURSUIT's 8.67 ms median is 32.9× faster than BGE-v2-m3's 285.612 ms median in full-batch reranking.The comparison scores all 200 candidates per question in one batch.Source: Table 13 · p. 28, §C.7 · p. 28
  • Figure 9 plots median reranking latency against batch size on logarithmic axes, with MEMPURSUIT shown as a batch-independent 8.67 ms reference.BGE sweeps extend to batch size 200 and Qwen sweeps to 144.Source: Figure 9 · p. 28

C.7.1 LATENCY ACROSS CANDIDATE AND CONTEXT BUDGETS

  • Across P=100–500 candidates and K=10–100 selected memories, median latency stays below 40 ms; at P=500 and K=100 it is 38.58 ms.At K=20, increasing P from 100 to 500 raises median latency from 6.30 ms to 22.47 ms.Source: §C.7.1 · p. 28, Table 14 · p. 29
Loading 2610.15642v1…