What Should an Agent Remember? Disentangling Retention from Retrieval in Bounded-Memory Evaluation

1 Introduction

• Memory architectures encode assumptions that recent, repeated, or query-resembling observations are worth keeping.With a budget of K items, the operational question is which observations remain available later.Source: §1 · p. 1
• Retention is decided as observations arrive without knowledge of the future query, while selection occurs at query time and can be query-aware.A single accuracy score cannot distinguish the effects of these stages from differences in what material methods can access.Source: §1 · p. 1
• An audit found that five earlier conditions differed in history access, query informativeness, and distractor design, producing a large but misleading result.Two conditions retained the full history but were reported far apart, a gap that could not reflect memory-budget behaviour.Source: §1 · pp. 1–2
• The rebuilt streaming-recall benchmark crosses retention and selection rules, evaluates every cell on the same seeded episodes, and reports recall with retention rate.The paper also decomposes the mixed-cell advantage, tests delay-induced eviction and benchmark artifacts, and repeats the ceiling analysis on SQuAD.Source: §1 · p. 2

2 Related work

• Agent-memory systems combine signals or mechanisms such as recency, importance, relevance, bounded context, external storage, and forgetting schedules.Examples include Park et al. [2023], Packer et al. [2023], and Zhong et al. [2024].Source: §2 · p. 2
• Combining memory-management decisions in end-to-end evaluations makes retention and retrieval effects hard to separate.This is the evaluation regime the paper targets.Source: §2 · p. 2
• Classical retrieval compares lexical and dense methods on a fixed corpus, whereas bounded memory can discard items before retrieval.The paper contributes to evaluation rather than proposing a memory architecture.Source: §2 · p. 2

3 A factorial streaming-recall benchmark

• An episode contains 10 facts followed by 2 queries, and each query targets a uniformly sampled non-distractor fact.Facts share the form “The slot is value”; slots are unique within an episode, while distractors use a disjoint slot vocabulary.Source: §3 · p. 2
• Retention rules operate online under capacity K, while selection rules rank survivors and expose the top k=3.The grid includes recency, random, frequency, privileged type-oracle, and unbounded retention; selection includes recency, random, query-similarity, dense, and oracle.Source: §3 · p. 2
• Informative queries name the target slot, whereas uninformative queries omit usable target information and differ by exactly one token.The informative form is “What is the slot ?”; the uninformative form is “What is the value?”, where value appears in no fact.Source: §3 · pp. 2–3
• The 100% entries are a construction ceiling: informative queries token-match exactly one fact, making selection exactly solvable when nothing was discarded.Table 1 reports required-fact recall under informative queries with K=5 and k=3.Source: Table 1 · p. 3
• Required-fact recall is the fraction of queries with the target among exposed items, and retention rate is the fraction whose target survived and upper-bounds recall.All cells use 300 seeded episodes; bootstrap confidence intervals resample episodes because queries from one stream are not independent.Source: §3 · p. 3

4.1 RQ1: how much of the advantage is query awareness?

• In every bounded row, QUERY, DENSE, and ORACLE reach the retention rate; across all 50 cells, no selection rule exceeds its row’s retention rate.A 768-dimensional MPNet retriever changes recall by +0.00 pp [+0.00, +0.00] against both lexical and oracle selection under bounded and unbounded retention; saturation is specific to this generator.Source: §4.1 · p. 3
• At matched access, query-aware selection adds +15.5 pp; the mixed comparison adds +68.7 pp, including a further +53.2 pp from full-history access.The matched-access gain is +15.5 pp [+12.8, +18.2], while the mixed comparison is +68.7 pp [+65.2, +72.0] and the access contribution is +53.2 pp [+49.7, +56.8].Source: §4.1 · p. 3, Figure 1 · p. 4

4.2 RQ2: behaviour under memory pressure

• Recency recall rises from 22.7 to 100 across K=2, 3, 5, 10, and 20, but falls to 0.0% at delays of 6–9 facts; type-oracle retention reaches 31.2%.Selection is inert when K≤k=3, and is oracle-equivalent across delay buckets; unbounded retention remains at 100%.Source: §4.2 · pp. 3–4, Table 2 · p. 4, Table 3 · p. 5

4.3 RQ3: benchmark properties that create spurious apparent gains

• With uninformative queries, QUERY exactly matches RECENCY; marking distractors boosts QUERY by +9.8 pp unbounded and +9.0 pp bounded.The scorer detects the literal marker rather than matching the query; other rules are unchanged.Source: §4.3 · p. 4, Table 6 · p. 6

4.2 RQ2: behaviour under memory pressure

• As distractor density rises from 0.1 to 0.7, type-oracle recall rises from 54.0 to 100.0 while recency remains roughly unchanged at 47.3 to 50.5.The type-oracle benefits because filtering more of the stream leaves proportionally more room for targets.Source: §4.2 · p. 4, Table 4 · p. 5

4.3 RQ3: benchmark properties that create spurious apparent gains

• With uninformative queries, DENSE reaches 42.0% by filtering distractors through latent type structure, not by extracting query signal.It exposes no distractors against a 30% base rate; selecting among the seven relevant facts predicts 42.9%, close to the 42.0% observed.Source: §4.3 · p. 5

4.4 RQ4: does recall predict answer correctness?

• With FLAN-T5-large, a bounded agent recalls the target for 47.3% of queries and answers 36.3% correctly; conditional answer accuracy spans 70.7–74.7%.No-memory accuracy is 0.0%; under these tested conditions, answer accuracy is recall scaled by a roughly constant factor.Source: §4.4 · p. 5

4.5 RQ5: what explains the remaining failures?

• All 319 failures of the bounded recency agent are evictions; displacement and similarity confusion account for zero failures.Using a query-independent selector on the same streams adds 93 displacement failures.Source: §4.5 · p. 5

5 Does this survive real text?

• The evaluation reruns the unchanged retention and selection grid, metric, and bootstrap on 300 SQuAD v1.1 dev questions, using answer-containing sentences as targets and unrelated sentences as distractors.Queries are real SQuAD questions rather than templates.Source: §5 · p. 6
• Across all 25 SQuAD cells, no selection rule exceeds its row’s retention rate; matched-access query awareness is worth +20.3 pp [+15.7, +25.0].Under bounded recency, dense retrieval matches the oracle selector at +0.00 pp [+0.00, +0.00], while unbounded access is worth +44.3 pp [+38.3, +50.3] to it.Source: §5 · p. 6
• Dense retrieval beats Jaccard by +6.0 pp [+3.0, +9.3] with unbounded retention and +1.7 pp [+0.3, +3.3] under a bounded buffer.The synthetic dense-versus-lexical equivalence reflects that generator’s exact query-token match, not a general result.Source: §5 · p. 6

6 Limitations

• Both environments test single-turn recall, not multi-session agent memory, so the measured results do not estimate memory behavior across sessions.The main environment is synthetic and small, with 10 facts and 2 queries per episode.Source: §6 · p. 6
• Answer-accuracy results use FLAN-T5-large with greedy decoding and one prompt, limiting the check to whether recall tracks correctness in this setting.Table 7 reports normalized exact match over 300 queries per condition.Source: §6 · p. 6, Table 7 · p. 7
• On SQuAD v1.1 dev, the retention ceiling held in all 25 cells, while DENSE beat QUERY but remained bounded by that ceiling under bounded retention.The evaluation used 300 questions with K=5 and k=3.Source: Table 8 · p. 7
• The positional delay measure does not capture elapsed time or cross-session persistence, so the setting is bounded streaming memory rather than long-term memory.A cross-encoder reranker could differ from the two scorers evaluated.Source: §6 · p. 7

• When required memory has been evicted, retrieval performance is upper-bounded by retention because a better retriever cannot recover it.Source: §7 · p. 7
• Comparisons should hold access fixed and report the retention rate beside every recall number.Source: §7 · p. 7
• One accuracy number cannot distinguish retention decisions from selection decisions.Source: §7 · p. 7