PaperlayerView exact revision on arXiv

Source-linked AI summary

The Retriever Should Remember: Experience-Amortized Reranking for Long-Term Agent Memory

Qi Feng, Chris Ding, Jicong Fan

arXiv:2608.22767v1cs.AI

TL;DR

Long-term agents accumulate memories but not the retrieval experience needed to identify query-relevant evidence efficiently. EARM preserves sparse LLM relevance scores in a causal query–memory matrix, completes missing scores, and mixes them with newly observed anchors; mixed rankings improve answer accuracy over semantic retrieval by up to 6.62% and remain effective at a 17.5% direct-scoring ratio.

  • Problem

    Long-term agents’ retrievers repeatedly score candidate memories because semantic similarity can miss query-conditioned relevance and stateless reranking discards prior scores.

  • Method

    EARM stores sparse LLM query–memory relevance scores in an online causal matrix, learns shared structure through completion, and combines observed and estimated scores for reranking.

  • Results

    2.14–6.62%: mixed observed-and-completed rankings improve answer accuracy over semantic retrieval, while remaining effective at a 17.5% direct-scoring ratio.

  • Takeaways & Limitations

    Retrieval experience can become a learned capability over an agent’s lifetime, allowing completed-score-selected memories to add gains beyond directly scored anchors.

  • Takeaways & Limitations

    EARM assumes reusable low-dimensional relevance structure, uses a predetermined budget schedule, and may propagate noisy or miscalibrated LLM scoring errors.

Abstract

from arXiv · show

Long-term language-model agents accumulate memories across interactions, but their retrievers typically do not accumulate retrieval experience. Semantic retrieval is efficient, but embedding similarity does not always reflect whether a memory contains evidence relevant to the current query. Large language model (LLM) rerankers provide stronger query-conditioned relevance scores, yet stateless reranking repeatedly scores a large candidate pool and discards these scores after each query. We introduce EARM, an experience-amortized reranking framework that treats previously acquired LLM relevance scores as reusable retrieval experience. EARM stores sparse query--memory relevance scores in an online matrix, learns their shared structure through causal matrix completion, and combines a small set of newly observed scores with estimated scores to rerank the remaining candidates. The scoring budget decreases as experience accumulates, changing LLM reranking from a repeated per-query expense into a retrieval capability learned over an agent's lifetime. Experiments on long-term conversational memory show that mixed observed-and-estimated reranking improves answer accuracy over semantic retrieval by up to 6.62% and remains effective when only 17.5% of candidates receive direct LLM relevance scores, thereby substantially reducing the inference overhead of LLM reranking. These results motivate a broader view of agent memory: a long-lived agent should remember not only past content, but also how that content has proved useful for retrieval.

I. INTRODUCTION

EARM addresses the mismatch between growing agent memories and stateless retrieval by preserving LLM relevance scores as reusable experience. It combines sparse direct scoring with completed scores to improve answer accuracy while reducing repeated reranking effort.

  • Long-lived agents need retrieval to surface stored experiences that provide evidence for the current query.
  • Embedding similarity efficiently filters candidates but does not always represent query-conditioned evidential relevance.Relevant memories may supply temporal, causal, or entity relations despite appearing semantically distant.
  • Stateless LLM reranking repeatedly scores candidates and discards the resulting relevance evidence across queries.For T queries with N candidates, pointwise reranking may require up to TN scoring operations.
  • 2.14–6.62%: mixed observed-and-completed rankings improve answer accuracy over semantic retrieval across evaluated settings.The method approaches full LLM reranking to within 2.34–2.79% in the strongest completion configurations.
  • EARM stores sparse LLM scores in an online query–memory matrix, learns shared structure through causal low-rank completion, and mixes observed with estimated scores.A small set of anchors positions each new query within a learned latent memory accessibility map.
  • 17.5%: EARM remains effective when only this direct-scoring ratio is used, while completed-score-selected memories add gains beyond observed anchors.The introduction reports up to 6.62% improvement over semantic retrieval and additional gains from completed scores.

II. RELATED WORK

Prior work develops persistent memory, memory organization, trajectory reuse, and LLM reranking, but EARM specifically reuses pointwise query–memory relevance scores to predict unseen pairs without task rewards.

  • Long-term memory research spans extraction, consolidation, compression, hierarchical organization, efficient access, and associative memory structures.
  • Other agents reuse past trajectories through reflections, transferable lessons, or reusable workflows for subsequent tasks.
  • EARM differs from utility-based approaches by storing pointwise LLM relevance scores for query–memory pairs rather than assigning one scalar utility to each memory.Its prediction target is unseen query–memory relevance, not task reward or a learned Q-value.
  • A first-stage semantic retriever returns a fixed-size candidate set, after which sparse LLM scoring estimates unscored candidates for sequential reranking.The setting permits Bt < N relevance-scoring operations and selects K memories for answer context.
  • The formulation assumes stable memory identities and recurring query–memory relevance structure so past scores can inform future rankings.The paper notes that these assumptions can fail in some settings.

IV. EXPERIENCE-AMORTIZED MEMORY RERANKING

EARM is a lifetime-developing retrieval pipeline that preserves candidate identities and combines semantic candidate generation with experience acquisition, matrix completion, and mixed reranking.

  • EARM comprises semantic candidate generation, experience acquisition, a causal retrieval-experience matrix, online low-rank completion, and mixed reranking.The complete pipeline runs from query arrival through answer evaluation.
  • Semantic retrieval computes embedding similarity and returns the top N memories as a deliberately high-recall candidate pool.This stage narrows the full store without determining the final context.
  • Persistent candidate identities ensure that a memory retrieved repeatedly occupies the same matrix row across queries.

B. Retrieval Experience Matrix

The retrieval-experience matrix records sparse LLM relevance history across persistent memories and queries, while staged anchor sampling supplies observations for online completion.

  • B. Retrieval Experience Matrix: The matrix contains one column per query, one row per persistent memory, and observed entries containing acquired LLM relevance scores.
  • B. Retrieval Experience Matrix: Unobserved cells distinguish retrieved-but-unscored memories from memories omitted by semantic retrieval, while pre-insertion cells remain unavailable.
  • C. Stage-Wise Experience Acquisition: The scoring budget is organized into experience stages, beginning with a larger observation ratio and decreasing as relevance structure accumulates.The schedule is predefined by query index rather than adapted to uncertainty or distribution shift.
  • C. Stage-Wise Experience Acquisition: Anchor sampling splits the candidate pool between high-similarity and lower-ranked regions to balance exploitation with exploration.The equal split increases matrix coverage and avoids observing only the semantic retriever’s preferred examples.

D. Causal Relevance Completion

EARM completes missing query–memory relevance scores with a causal, bias-aware low-rank model that uses historical observations and current-query anchors. The completed scores can rerank omitted candidates, while retained observations refine later queries and amortize reranking effort over time.

  • The incomplete retrieval-experience matrix contains direct scores only for each query's anchor subset.
  • A bias-aware low-rank model combines global, memory-specific, and query-specific biases with latent memory–query factors to estimate missing scores.
  • Historical observations learn shared memory parameters, while current anchors constrain new query parameters before estimating scores for unobserved candidates.
  • The same completion model can score historically supported memories omitted by semantic retrieval, expanding the ranking domain beyond the current candidate set.
  • Observed anchors remain in the matrix and refine shared parameters for later queries, amortizing reranking effort across time.
  • Causal masking prevents query t from using scores from later queries, producing an evolving accessibility map rather than an offline reconstruction.

E. Mixed Reranking and Context Construction

EARM ranks candidates with a mixed policy that preserves direct LLM scores and fills remaining decisions with completed scores. Selected memories form the answer context and are then restored to chronological order for coherent generation.

  • The experiments rank within the semantic candidate set, while the expanded form can include completion-scored memories outside that set.
  • The highest-scoring K memories enter the answer context, after which selected memories are restored to chronological order.
  • Direct LLM relevance scores remain authoritative where available, while completed scores compete for the remaining context slots.

V. EXPERIMENTAL DESIGN

The evaluation tests EARM against semantic retrieval, full LLM reranking, and observed-only scoring on long-term conversational memory. It varies direct-scoring budgets across experience stages to assess accuracy, efficiency, and the contribution of completion.

  • The evaluation asks whether experience-amortized reranking improves different question types over semantic retrieval and approaches full LLM reranking.
  • LoCoMo supplies long conversations and long-term memory questions, using Mem0 for memory extraction and storage while modifying retrieval and reranking.
  • Each query returns N = 200 semantic candidates, and the relevance scorer assigns scores in [0, 1] to query–memory pairs.
  • Direct-scoring budgets decrease from 200 to 35 candidates across stages, corresponding to observation ratios from 100% to 17.5%.
  • The four policies are Semantic, Full LLM reranking, Observed-only, and Mixed (EARM).

D. Metrics and Statistical Analysis

The study measures end-to-end answer accuracy across question types and analyzes how completion contributes beyond directly scored memories. It also counts direct query–memory scoring calls as the reranking cost.

  • The primary metric is LLM-evaluated answer accuracy, reported overall and for multi-hop, open-domain, single-hop, and temporal questions.
  • Reranking cost is counted as one LLM call for each directly scored query–memory pair.
  • The comparison spans semantic retrieval with no LLM scores and full LLM reranking that scores every candidate.
  • The study tests whether completion remains better than semantic retrieval as the direct-scoring ratio falls from 100% to 17.5%.
  • Observed-only minus Semantic measures directly scored memories' contribution, while Mixed minus Observed-only measures the additional contribution of completed-score selections.

VI. RESULTS

EARM improves answer accuracy over semantic retrieval while using substantially fewer LLM reranker calls, and remains effective as direct scoring becomes sparse. Completion contributes additional gains beyond directly scored memories alone.

  • Overall and Question-Type Accuracy: 6.62%: Rank-8 EARM raises Top-10 overall accuracy from 82.21% to 88.83% and comes within 2.79% of full LLM reranking.For Top-20 retrieval, rank-2 EARM improves overall accuracy by 2.99% and remains 2.34% below full reranking.
  • Overall and Question-Type Accuracy: 74.43% fewer LLM reranker calls: EARM uses 78,736 calls versus 307,982 for full reranking.The best EARM configuration recovers approximately 70% of full reranking's improvement at Top-10 and 56% at Top-20.
  • Accuracy Across Experience Stages: 5.53%: At a 17.5% direct-scoring ratio, rank-8 EARM improves Top-10 accuracy in B5+, while rank-2 improves Top-20 accuracy by 2.46%.Both completion ranks continue to outperform Semantic across context sizes as the observation ratio decreases.
  • Directly Scored versus Completed Memories: 0.78–2.79%: Completed scores add this improvement beyond directly scored memories alone in every configuration.At r = 8 and Top-10, completed-score-selected memories add 2.79% on top of the 3.83% gain from directly scored memories.

VII. DISCUSSION

EARM distinguishes content memory from retrieval memory: it preserves evidence about which memories help which information needs and transfers that experience across distinct queries. Matrix completion can also estimate scores for persistent memories outside the current semantic candidate set, expanding retrieval beyond the initial pool.

  • From Content Memory to Retrieval Memory: Retrieval memory records which stored memories have proved useful for particular information needs, complementing content memory about the agent's experiences.EARM turns otherwise disposable LLM relevance scores into persistent state for future retrieval.
  • From Content Memory to Retrieval Memory: EARM transfers retrieval experience across distinct queries through shared memory biases, query biases, and latent interactions rather than requiring exact query repetition.Candidate overlap helps, but a new query can benefit when sampled anchors connect it to previously learned structure.
  • From Candidate Reranking to Search Expansion: Completed scores can admit historically supported memories absent from the current semantic candidate set, allowing temporal, causal, or complementary evidence into context.Fixed-pool experiments isolate reranking, whereas ranking completed estimates with semantic candidates expands the search domain.

D. What Must Be True for Amortization to Work

Amortization depends on persistent cross-query structure in query–memory relevance, while the framework is bounded by fixed budgets, low-dimensionality assumptions, and noisy pointwise LLM scores. Experiments nevertheless report gains at sparse direct-scoring ratios and from completed-score-selected memories.

  • What Must Be True for Amortization to Work: Amortization requires persistent cross-query relevance structure, such as recurring user interests, stable memory importance, repeated entities, or common information needs.Unrelated memory subsets, unstable identities, or rapidly changing memory meanings provide little leverage from historical scores.
  • Limitations: The framework uses a predetermined budget schedule, assumes reusable low-dimensional structure, and can propagate noisy, miscalibrated, or prompt-sensitive reranker errors.Highly heterogeneous query streams can violate the matrix-completion assumption.
  • Empirical Evidence: Completed-score-selected memories provide additional gains beyond directly scored anchors, supporting the value of learned retrieval experience.The conclusion reports this effect alongside improved end-to-end answer accuracy over semantic retrieval.
Loading 2608.22767v1…