PaperlayerView exact revision on arXiv

Source-linked AI summary

Agent Memory as Recommendation: Recommender-System Design for Long-Term Memory of LLM Agents

Xinyu Zhang, Weiwei Sun

arXiv:2610.34673v1cs.IRcs.AIcs.CL

TL;DR

LLM agents must select complete evidence from large, growing memory stores under per-request context and cost budgets, while existing memory designs leave component contributions unclear. This paper frames memory as budgeted slate recommendation and proposes AgentMemRec, a no-LLM-call selector. It reports stronger multi-session results and lower serving cost, with gains associated with more complete evidence slates.

  • Problem

    Agents must select all evidence needed for multi-session questions from large, growing stores under limited context and per-request cost budgets, while separately designed memory systems obscure which components yield accuracy.

    Source: §1 · p. 1

  • Method

    The paper formalizes memory selection as budgeted slate recommendation and builds AgentMemRec with cheap recall channels, a learned ranker, and slate construction, without LLM calls.

    Source: §1 · p. 1

  • Results

    On multi-session LongMemEval, AgentMemRec beats memory and context selectors by 10 to 31 accuracy points, matches or exceeds a 64-call reranker at 3.2 times lower cost, and outperforms Mem0 and Zep on LoCoMo.

    Source: §1 · pp. 1–2, §8 · p. 8

  • Takeaways & Limitations

    AgentMemRec slates hold all evidence for 0.94 of questions, and recommender-system design offers an effective and efficient foundation for long-term agent memory.

    Source: §1 · p. 2, §8 · p. 8

    The item-wise ranking and slate-filling rationale depends on reward conditions; composition independence holds only approximately for incomplete slates and begins to fail at κ≥3.

    Source: §7 · p. 8

Abstract

from arXiv · show

Large language model (LLM) agents that serve users across many sessions must choose, for every request, the few stored memories that enter a limited context window. This choice largely decides whether the agent answers correctly, and it runs on every request at Web scale. This selection faces three challenges: a large and growing store, a per-request cost budget, and multi-session questions that need all of their evidence. Many memory systems build notes or graphs with hundreds to thousands of LLM calls per history, and each is designed on its own, so it is unclear which component yields the accuracy. These challenges mirror the properties of recommendation, so we formalize agent memory as budgeted slate recommendation, with the request as the user, stored memories as items and the context window as a slate. The formalization places memory systems in one design space of recommender cascades and factorizes accuracy into the completeness that each stage controls. Guided by it, we propose AgentMemRec, which adapts candidate generation, learning to rank and slate construction from recommender systems and selects memories without any LLM call. On multi-session questions of LongMemEval, AgentMemRec outperforms memory and context-selection methods, including A-Mem, HippoRAG 2, DynamicRAG and T-Mem, by 10 to 31 points of accuracy. It matches an LLM reranker that makes 64 calls per query at 3.2 times lower serving cost, and on LoCoMo it outperforms Mem0 and Zep without any LLM call at write time. The factorization attributes the gains to slates that more often hold all of the evidence, so recommender-system design offers an effective and efficient foundation for the long-term memory of LLM agents.

1 Introduction

Long-term memory selection is a recurring, budgeted decision: the agent must retrieve all evidence needed from a large store for each request. The paper formalizes this as budgeted slate recommendation and introduces AgentMemRec, a no-LLM-call selector that improves multi-session results.

1 Introduction

  • Selection is difficult because histories grow to about 500 sessions and 1.5M tokens while the answering model reads at most 2,700 tokens, and multi-session questions require all evidence.Missing one evidence turn can cost 37 to 64 accuracy points with Qwen in the compared turn-level designs.Source: §1 · p. 1
  • Existing memory systems use varied structures and often hundreds to thousands of LLM calls, while their separate designs obscure which component yields accuracy.For HippoRAG 2, A-Mem, and T-Mem on LongMemEval-S, building memories takes about 440 to 2,270 calls per question; compared turn-based methods hold all evidence for only 0.50 to 0.68 of multi-session questions.Source: §1 · p. 1
  • The paper casts requests as users, stored turns as items, and the bounded context as a slate, placing memory systems in a shared design space of recommender cascades.The design objective is to raise pool and ranking completeness within an LLM budget and then fill the slate.Source: §1 · p. 1
  • AgentMemRec uses two cheap recall channels, a ranker learned over ten cheap features, and pool sizing based on the product of pool and ranking completeness.Its slates hold all evidence for 0.94 of questions, and systems do not differ significantly on jointly complete slates.Source: §1 · pp. 1–2
  • On multi-session LongMemEval-S, AgentMemRec beats Adaptive-k, DynamicRAG, HippoRAG 2, A-Mem, and T-Mem by 10 to 31 accuracy points; on LongMemEval-M, gains are 11 to 33 points.It matches or exceeds a reranker making 64 LLM calls per question at 3.2 times lower serving cost, and outperforms Mem0 and Zep on LoCoMo without an LLM call at write time.Source: §1 · pp. 1–2

2 Related Work

Prior work builds long-term memories with varied representations and studies budgeted context selection, while recommender research offers techniques for candidate generation, ranking, and slate decisions. The paper positions AgentMemRec as read-time context selection over raw turns, without LLM calls.

2 Related Work

  • Long-term memory systems store linked notes, consolidated facts, temporal graphs, fact graphs, hypergraphs, hierarchies, segments, or antecedent links; verbatim chunks have also outperformed extracted artifacts on long conversations.In the recommender view, these designs function as item side-information or recall structure, and most spend LLM calls at write time.Source: §2 · p. 2
  • Budgeted context selectors include Adaptive-k, which cuts at a similarity-score gap, and DynamicRAG, which learns passage order and quantity; SetR uses LLM-based set selection.Prior work also reports harms from high-scoring non-answering passages, hard negatives, and input length alone.Source: §2 · p. 2
  • Recommender and search systems separate candidate generation from ranking, use cascades and set-aware stages, and provide tools such as LambdaRank, reciprocal rank fusion, and LLM-ranker distillation.AgentMemRec formalizes memory as budgeted slate recommendation and selects a context, unlike RMM's online feedback from answer citations or RecMem's consolidation decision.Source: §2 · p. 2

3 Agent Memory as Recommendation

The section models long-term memory selection as budgeted slate recommendation and uses a recommender cascade to organize memory designs and their accuracy drivers.

3.1 Memory selection as slate recommendation

  • Memory selection treats each question and agent state as a request, history turns as catalog items, rendered token count as cost, and answer acceptance as slate reward.The selected slate must fit the token budget for every question.Source: §3.1 · p. 2

3.2 The design space

  • A memory design is a cascade of write-time mapping, candidate generation, ranking, and budgeted slate construction, with LLM cost counted across those stages.Compared systems occupy points in this design space and differ mainly in where they spend LLM calls.Source: §3.2 · p. 3

3.3 Three results about the space

  • Accuracy factorizes into incomplete-slate reward plus the complete-versus-incomplete reward gap multiplied by pool and ranking completeness.Candidate generation sets pool completeness, while ranking sets ranking completeness; set-level effects act through rewards on complete and incomplete slates.Source: §3.3 · p. 3
  • With slate-invariant rewards, additive costs, and independent evidence units, selection reduces to a knapsack with values −log(1−p_v), or highest-probability units at equal costs.Slate composition effects or correlated evidence are conditions under which item-wise ranking may stop being sufficient.Source: §3.3 · pp. 3–4
  • When reward does not decrease as a fitting prefix is extended, greedy fill is at least as good as a cutoff; the fill can skip turns that do not fit.The fill need not be a ranking prefix, and a cutoff can help only when some added unit lowers slate value.Source: §3.3 · p. 4

3.4 The design problem

  • The design objective prices pool completeness, ranking completeness, and LLM budget together, while a wider pool raises completeness but also gives the ranker more distractors.AgentMemRec applies this framework with a wide cheap pool, a learned ranker over cheap features, pool sizing by the completeness product, and budget filling.Source: §3.4 · p. 4

4 AgentMemRec

AgentMemRec applies the cascade with no LLM calls at selection time, using retrieval, learned ranking, and budget filling; an optional extension adds write-time records.

4 AgentMemRec

  • AgentMemRec uses candidate generation to raise pool completeness, learning to rank to raise ranking completeness, sizes the pool by their product, and fills the budget without selection-time LLM calls.Its three selection stages instantiate the design objective with an LLM budget of zero.Source: §4 · p. 4

4.1 Candidate generation: raising 𝑟pool

  • A 128-turn pool from BM25 and bge-m3 retrieval merged by reciprocal rank fusion reached 0.912 slate completeness product on development, versus 0.879 for 64 turns.Pool size was chosen by slate completeness after ranking.Source: §4.1 · p. 4

4.2 Learning to rank: raising 𝑟rank

  • A LightGBM LambdaRank ranker learns from gold labels on 19,456 question-turn pairs using ten cheap relevance and turn features.The training labels come from 152 development questions with LongMemEval-S and LongMemEval-M pools.Source: §4.2 · p. 4

4.3 Slate construction: the fill

  • The slate builder walks turns in rank order, adds each that fits the rendered-token budget, skips those that do not, and continues through the pool.The fixed budget is 2,700 tokens, evaluated with perturbations from 2,400 to 2,700; choosing among three budgets per question gained no more than decode noise.Source: §4.3 · p. 4

4.4 Optional write-time records

  • An optional write-time extension uses Qwen3-32B to create typed session records, then adds up to 32 record-cited turns to the 128-turn pool without selection-time LLM calls.Relative-time tags attach citing-record dates to turns containing relative time expressions, limited to dates no later than the question’s reference date.Source: §4.4 · p. 5

5 Experimental Setup

The experiments compare memory and context-selection systems on LongMemEval and LoCoMo under shared evaluation protocols, using paired comparisons and repeated slates to address decoding variability.

Benchmarks and splits

  • LongMemEval-S has 500 questions over about 50-session histories, LongMemEval-M uses the same questions over about 500 sessions, and LoCoMo has 1,540 scored questions in 10 conversations.The primary test stratum is κ≥2 with n=209; κ≥3 has 49 test questions.Source: §5 · p. 5

Answering and judging

  • LongMemEval answers use Qwen3-32B and DeepSeek V4.1 Flash with the official prompt and GPT-4o judging; LoCoMo uses T-Mem’s GPT-4o-mini answering and judging setup.A Qwen3-32B judge agrees with GPT-4o on 94% to 98% of answers and changes one primary label.Source: §5 · p. 5

Implementation details

  • Four slates per question and system average judge labels across budgets of 2,400 to 2,700 tokens because small slate changes can alter judged answers.Changing only the lowest-ranked 100 to 300 tokens changed the judged answer on 91 of 300 multi-session questions for Qwen and 43 for DeepSeek; paired percentile bootstrap uses 10,000 draws.Source: §5 · p. 5

Answering and judging

  • Every AgentMemRec margin over a memory or context-selection method is a gain, while comparisons with Ref are parity except a gain on LongMemEval-M with DeepSeek.Parity is non-inferiority at a 3-point margin with no significant difference.Source: §5 · p. 5

Compared systems

  • The shared-budget comparison covers context selectors, LLM-built memories, and a 64-call-per-question LLM reranker, with T-Mem the main LoCoMo comparison.The LongMemEval memory design is run on LongMemEval-M only; T-Mem is reproduced on LoCoMo and built by the authors on LongMemEval-S.Source: §5 · p. 5

6 Results

Across LongMemEval, AgentMemRec outperforms memory and context-selection methods, while slate completeness accounts for much of the advantage. It also matches the 64-call reranker at lower serving cost and performs competitively under shared LoCoMo protocols.

6.1 RQ1: Recommendation against LLM-built memory

  • 10.4 to 33.3 points: AgentMemRec outperforms the compared memory and context-selection methods on LongMemEval-S and LongMemEval-M, with gains across those comparisons.On LongMemEval-S, the margin is 10.4 to 31.2 points; on LongMemEval-M, it is 10.6 to 33.3 points.Source: §6.1 · pp. 5–6, Table 1 · p. 6

6.2 RQ2: Which terms of the factorization carry the gain

  • Slate completeness ranges from 0.50 for HippoRAG 2 and A-Mem to 0.94 for AgentMemRec, and orders accuracy across 11 designs with a Spearman correlation of 0.95.The factorization attributes most of the margins over turn-level methods to completeness; on jointly complete slates, systems do not differ significantly.Source: §6.2 · p. 6
  • The fused 128-turn pool holds all evidence for 0.981 of questions, yielding slate completeness of 0.940 against Ref’s 0.915.On Ref’s own pool, AgentMemRec’s ranker reaches Ref’s completeness, supporting the wider pool as the source of its advantage.Source: §6.2 · pp. 6–7

6.1 RQ1: Recommendation against LLM-built memory

  • At 𝜅≥3, AgentMemRec is the most accurate system for both answering models, and Figure 3 reports parity with the 64-call reranker without an LLM call.The margins over Adaptive-k, DynamicRAG, HippoRAG 2 and A-Mem grow to 23.0 to 44.9 points; accuracy is 68.4 and 77.6.Source: §6.1 · pp. 5–6, Figure 3 · p. 7

6.2 RQ2: Which terms of the factorization carry the gain

  • LambdaRank raises ranking completeness from 0.783 for the recall order to 0.959, with gains also present when evidence spans at least three sessions.The ranker combines relevance and turn-context features; a ranker distilled from Qwen3-8B scores reaches 0.933 without gold labels.Source: §6.2 · p. 7
  • At half the budget, AgentMemRec remains above every memory and context-selection method at 2,700 tokens; optional relative-date tags help LoCoMo temporal accuracy but can hurt LongMemEval-M.The tags raise LoCoMo temporal accuracy by 7.2 points and lower LongMemEval-M temporal reasoning for DeepSeek by 8.2 points.Source: §6.2 · p. 7

6.3 RQ3: Cost

  • AgentMemRec costs 3.2 times less per request than the 64-call reranker at the same accuracy and makes no LLM calls at write time.At saturation, selection takes 280 GPU-milliseconds per request against 891 for Ref.Source: §6.3 · p. 7

6.1 RQ1: Recommendation against LLM-built memory

  • Under shared protocols, AgentMemRec scores 82.5 on LongMemEval-S test questions and 83.2 on all 500, and reaches parity with the T-Mem reproduction on LoCoMo.On LoCoMo, it scores 77.0 against 77.4 for the reproduction and above Mem0 and Zep.Source: §6.1 · p. 6, Table 2 · p. 8

7 When Item-Wise Ranking Suffices

The item-wise reduction depends on reward behavior across slates. Its assumptions hold best when evidence is complete, but near-miss filler and highly scattered evidence expose limits.

7 When Item-Wise Ranking Suffices

  • The item-wise reduction assumes slate accuracy does not depend on composition and that adding a unit cannot lower reward.The paper tests these conditions on evaluated and label-controlled slates.Source: §7 · p. 8
  • At 𝜅≥3, jointly complete AgentMemRec slates score 8.7 points below cross-encoder order for Qwen and 6.5 below for DeepSeek; near-miss filler can lower reward.The filler costs vary by answering model, and the authors note that negative item values could also explain them.Source: §7 · p. 8
  • Along AgentMemRec’s LongMemEval-S ranking and LLM-ranked LongMemEval-M slates, reward does not fall beyond decode noise, but the assumptions fail at 𝜅≥3 and with near-miss filler.The paper identifies set-level objectives as a next recommender design to test in those settings.Source: §7 · p. 8

8 Conclusion

The paper frames long-term agent memory as budgeted slate recommendation and uses recommender-system principles to design AgentMemRec. Its conclusion is that the approach offers an effective and efficient foundation within the evaluated scope.

8 Conclusion

  • The paper formalizes selecting stored memories for a bounded context as budgeted slate recommendation and places memory systems in a shared design space of recommender cascades.Its factorization reduces memory design to raising each stage’s completeness within an LLM budget.Source: §8 · p. 8
  • AgentMemRec selects memories without LLM calls and outperforms LLM-built memories and context selectors on LongMemEval, while matching or exceeding an LLM reranker at a fraction of the cost.On LoCoMo, it outperforms production memory layers.Source: §8 · p. 8
  • The reported gains come from slates that more often hold all of the evidence; set-level objectives, models of user state and learning from use are identified as next steps.Source: §8 · p. 8

A Data, Rendering and Judging

The experiments use specified LongMemEval and LoCoMo releases, rendering rules, and answering and judging protocols. Repeated-answer agreement and judge calibration provide context for interpreting measured accuracy.

A Data, Rendering and Judging

  • The cleaned LongMemEval release of 2025-09 contains LongMemEval-M and -S with about 4,894 and 490 turns per question, respectively, and shared 500 question identifiers.The frozen split includes answerable and unanswerable test questions; every answerable question has a gold turn.Source: A Data, Rendering and Judging · p. 10
  • LoCoMo contains 10 conversations of 369 to 689 turns and 1,540 scored questions, including 332 questions with evidence in at least two sessions.The split uses conversations 0–2 for development and 3–9 for test.Source: A Data, Rendering and Judging · p. 10
  • Selected LongMemEval turns are grouped by session and ordered by date, while LoCoMo slates list selected turns under dated session headers.Token counts use the o200k tokenizer; the answering and judging setup uses Qwen3-32B and a GPT-4o judge on test questions.Source: A Data, Rendering and Judging · p. 10
  • Identical slates produce the same per-question answer 85.6% of the time for Qwen and 95.7% for DeepSeek, while repeated judging agrees 99.9%.The Qwen3-32B judge agrees with GPT-4o on 97.3% of 2,000 answers; the reference’s accuracy varies by 2.6 and 3.3 points across four near-identical budgets for Qwen and DeepSeek.Source: A Data, Rendering and Judging · p. 10

B Implementation Details

This section specifies the learned ranker and reports additional evidence on slate completeness, memory records, and evaluation conditions. Across evidence-session counts, AgentMemRec’s margins over memory and context-selection methods widen for questions requiring at least three sessions.

Selection implementation

  • The ranker is a 31-leaf LightGBM LambdaRank model trained on 304 development pools, using RRF-fused BM25 and bge-m3 rankings and a sigmoid cross-encoder score.RRF uses k=60; the ranker uses 300 trees and a 0.05 learning rate.Source: B Implementation Details · p. 10

Results by evidence sessions

  • At κ=2, AgentMemRec exceeds every memory and context-selection method by 6.1 to 27.0 points; at κ≥3, its margins over four baselines grow to 23.0 to 44.9 points.Table 5 reports accuracy by evidence-session count; AgentMemRec is the most accurate system at κ≥3.Source: Table 5 · p. 10, B Implementation Details · p. 10

Reference and evaluation conditions

  • The LLM-ranked reference scores each pooled turn from 0 to 10 with one call per turn; latency uses 50 sequential questions on one A100, while serving cost uses 209 test questions.The cost and latency measurements exclude the shared BM25 and dense pool.Source: B Implementation Details · pp. 10–11

Write-time records

  • Write-time records close a 2.3-point LoCoMo gap to T-Mem at its 3,800-token context; on 88 LongMemEval-M temporal questions, tags change accuracy through complete slates (−7.5 of −8.2 points).The tags displace 5 of 712 gold turn slots.Source: B Implementation Details · p. 10

Ranker feature ablation

  • Dropping turn-context features costs 13.4 points of slate completeness at κ≥2 and 31.1 at κ≥3, while dropping role, position, age, or length alone costs at most 2.9.Pool position with the four turn-context features alone reaches 0.903.Source: B Implementation Details · pp. 10–11, Table 4 · p. 10

Generalization and single-session questions

  • On 119 single-session test questions, AgentMemRec scores 89.5 and 94.3, at parity with Ref; on LoCoMo, a LongMemEval-trained ranker performs at parity with one trained on LoCoMo development conversations.Training on 128-turn rather than 64-turn pools changes slate completeness by at most 0.1 points.Source: B Implementation Details · p. 11

C Case Studies

The case studies show how learned ranking can place evidence turns within the slate budget when cheap recall orders them too low. The baseline-deviation section documents implementation choices and comparison protocols that qualify the reported results.

C Case Studies

  • In the concerts case, AgentMemRec ranks the five evidence turns 1, 2, 3, 6, and 7 and is the only system whose slate keeps all five.The recall order ranks them 1, 14, 38, 61, and 67; the cross-encoder ranks them 11, 16, 23, 36, and 47.Source: C Case Studies · p. 11
  • Across four multi-session cases, AgentMemRec’s learned ranker brings every evidence turn into the slate; Ref misses evidence in three cases despite making 64 LLM calls per question.The cases cover ordering concerts and sports events, counting festival mentions, and resolving an investment from a time reference.Source: C Case Studies · p. 11

D Baselines and Deviations

  • Baseline implementations differ from their official settings: DynamicRAG selects from the top 40 fused turns, while T-Mem uses Qwen3-32B as its builder on LongMemEval-S.The LongMemEval memory-design variant uses Qwen3-32B for extraction and bge-m3 for dense ranking; HippoRAG 2 uses one passage per round and one index per history.Source: D Baselines and Deviations · pp. 11–12
  • The selection-stage cost table reports that AgentMemRec makes no LLM call and needs about a third of the GPU time and latency of the 64-call reranker.Table 7 compares LongMemEval-M systems under GPT-4o with four perturbed slates per question.Source: Table 8 · p. 12, Table 7 · p. 12
  • Copied LoCoMo results use the T-Mem table’s reported Zep, Mem0, and MemOS values; the T-Mem reproduction scores 2.8 points below its reported value, while a same-day rerun is at parity (+0.4).HyperMem scores 72.6 with the shared reader and 87.5 with its own GPT-4.1-mini chain-of-thought reader; a conversation built with GPT-4.1-mini scores 92.1 versus the reported 92.7.Source: D Baselines and Deviations · p. 12

E Results on LongMemEval-M

On LongMemEval-M, AgentMemRec is at parity with Ref under the single-decode primary test and under GPT-4o with four perturbed slates. Label-controlled slates show that both completeness and filler choice relate to answer accuracy.

Main results

  • With GPT-4o and four slates, AgentMemRec is at parity with Ref for Qwen (+3.3) and exceeds it for DeepSeek (+5.1); it exceeds other systems by 10.6 to 33.3 points.The Qwen judge labels the DeepSeek comparison as parity; the LongMemEval design and HippoRAG 2 have slate completeness 0.495 and 0.342, versus 0.861 for AgentMemRec.Source: E Results on LongMemEval-M · p. 12

Primary test

  • In the primary single-decode test, AgentMemRec scores 68.9 and 79.9 against Ref’s 65.6 and 76.1, with gains of +3.3 and +3.8 and parity for both models.The averaged protocol is a replication of the single-decode runs.Source: E Results on LongMemEval-M · p. 12

Label-controlled slates

  • On label-controlled LongMemEval-M slates, the LLM ranker yields slate completeness 0.823 and Qwen and DeepSeek accuracy 0.690 and 0.780; gold-in-pool then cross-encoder order yields 0.850, 0.727, and 0.807.With gold turns fixed, ranked near-miss filler costs more than random filler of the same length for both answering models.Source: E Results on LongMemEval-M · p. 12

Related papers

Loading 2610.34673v1…