PaperlayerView exact revision on arXiv

Source-linked AI summary

When Retrieval Is Not Enough: Obsolete Facts in Agent Memory

Oleeviya Babu Poikarayil, Tony Basil Babu, Abdulrahman Nahhas, Het Darshan Mehta, Jawid Ahmad Baktash, Ernesto William De Luca, Klaus Turowski

arXiv:2610.36308v1cs.AI

TL;DR

Memory-agent benchmarks can conflate whether an update was retrieved with whether the model selected the currently valid version. This paper tests that distinction on FactConsolidation; an equal-size intervention finds that obsolete competitors harm both evaluated models.

  • Problem

    End-to-end accuracy can obscure whether failures arise from missing an update or not using a retrieved update when an older version is also present.

    Source: §1 · p. 1

  • Method

    The authors vary BM25 evidence budgets and compare frozen-prompt, equal-size four-statement contexts with and without an obsolete competitor on FactConsolidation.

    Source: §1 · pp. 1–2, §2 · p. 2

  • Results

    Qwen2.5 fell from 74/74 correct answers to 33, while Qwen3 fell from 100% to 87.8% when unrelated evidence was replaced by an obsolete competitor.

    Source: §3.2 · p. 3

  • Takeaways & Limitations

    Retrieving an update does not guarantee its use; the results motivate retiring, down-weighting, or marking superseded facts inactive while preserving provenance and validity information.

    Source: §4 · p. 4

    Evidence is limited to two Qwen models, deterministic decoding, one synthetic single-hop benchmark, and lexical retrieval.

    Source: §4 · p. 4

Abstract

from arXiv · show

Memory-agent benchmarks often collapse retrieval and answer generation into one score, making it hard to tell whether a failure comes from missing an update or from not using an update that was already retrieved. We examine this distinction on the single-hop FactConsolidation task of MemoryAgentBench. With Qwen2.5-7B-Instruct on 6K-history BM25 accuracy drops from 45% to 32% when the retrieved evidence budget grows from 64 to 512 tokens, even though a gold-assisted post-hoc audit finds the newest statement in every retrieved set. Most of the errors occur on questions that also contain an obsolete statement version. Under a frozen prompt and equal four-statement evidence size, replacing one unrelated statement with the obsolete competing statement reduces accuracy from 100% to 44.6% for Qwen2.5 and from 100% to 87.8% for Qwen3-14B, with all 50 harmful paired flips classified as stale. A 32K-history replication likewise favors BM25-64 over BM25-512 (69% versus 58%). These results suggest that retrieving the correct fact is not enough. An obsolete competing statement can be more harmful than an unrelated piece of evidence of the same size.

1 Introduction

The paper separates retrieving relevant information from selecting the currently valid version when memory contains an obsolete fact. It studies whether evidence budget and obsolete competitors affect use of retrieved updates.

1 Introduction

  • End-to-end accuracy hides whether an error reflects missing an update or choosing an older fact that was retrieved.A retriever may return the newest fact while the model still answers with the older one.Source: §1 · p. 1
  • The study asks how evidence budget, unrelated distractors, and an older competing fact affect use of a retrieved update.It uses FactConsolidation, where higher-numbered chronological statements replace lower conflicting ones, including synthetic facts that may contradict world knowledge.Source: §1 · pp. 1–2
  • Three experiments vary BM25 budget, compare questions with and without an obsolete competitor, and substitute that competitor for equal-size unrelated evidence.The tests examine whether obsolete target evidence is more damaging than unrelated evidence of the same size.Source: §1 · p. 2
  • Prior work covers multi-session memory, retrieval, distraction, contradictory evidence, and invalidated knowledge; this study adds a controlled intervention isolating evidence composition.The intervention holds the reader, prompt, latest fact, and four-statement size fixed, unlike the cited pipeline comparison that varied several factors.Source: §1 · p. 2

2 Experimental design

The experiments use MemoryAgentBench’s Conflict_Resolution split and BM25 retrieval, then compare budget sweeps with a frozen-prompt, equal-size intervention. The intervention changes evidence composition by replacing unrelated evidence with an obsolete target statement.

2 Experimental design

  • The study uses 100-question single-hop 6K and 32K FactConsolidation histories, with 74 conflict and 26 no-conflict questions in the 6K set.The 6K and 32K histories contain 455 and 2,310 statements, respectively; primary results use normalized substring scoring.Source: §2 · p. 2
  • BM25 ranks statements to token budgets, restores chronology, and is tested at 64, 128, 256, and 512 tokens for 6K histories and at both endpoints for 32K.A gold-assisted audit checks whether the latest statement is present; it is diagnostic, not a deployable retrieval procedure.Source: §2 · p. 2
  • For each of 74 conflict questions, two four-statement contexts differ only by replacing one unrelated statement with the obsolete competitor.Distractors have zero question content-word overlap, and a frozen prompt protects the paired contrast; these questions were also used for prompt development and the pilot.Source: §2 · p. 2
  • The final controlled experiments use Qwen2.5-7B-Instruct and Qwen3-14B with deterministic decoding and a frozen prompt specifying chronological replacement.Paired exact McNemar tests are treated as benchmark diagnostics because questions share a history.Source: §2 · p. 2

3 Results

Accuracy fell as BM25 budgets grew even when the latest statement was retrieved, and failures concentrated on conflict questions. In an equal-size intervention, obsolete competitors harmed both models; the same budget direction appeared in the 32K replication.

3.1 Budget growth exposes a conflict-resolution failure

  • 45% to 32%: 6K BM25 accuracy fell as the budget grew from 64 to 512 tokens, despite the latest statement appearing in every retrieved set.Conflict-question accuracy fell from 25.7% to 12.2%, compared with 100% to 88.5% for no-conflict questions; conflict status is observational.Source: §3.1 · p. 3

3.2 An obsolete fact is more harmful than an unrelated fact

  • Qwen2.5 answered all 74 conflict questions correctly in condition A but only 33 in B; Qwen3 fell from 100% to 87.8%, with harmful flips selecting stale values.Condition B replaces one unrelated statement with the obsolete competitor while retaining the latest statement.Source: §3.2 · p. 3
  • The Figaro example shows models switching from the newer “Thomas Kyd” to the obsolete “Pierre Beaumarchais” despite the newer statement and chronology prompt remaining visible.This illustrates stale selection in the equal-size intervention.Source: §3.2 · p. 3

3.3 32K replication and manual audit

  • 69% versus 58%: on the 32K history, official scores favored BM25-64 over BM25-512, and all 18 harmful paired flips were correct-to-stale.Manual inspection found two ambiguous cases among automatically labeled no-conflict questions; excluding them for sensitivity gave 70.4% versus 58.2%.Source: §3.3 · p. 4
  • Manual metric review found both false negatives and false positives, while representative audited outputs use the later synthetic memory—not ordinary world knowledge—as truth.Two final Qwen2.5 outputs contained both latest and stale strings.Source: §3.3 · p. 4, Table 2 · p. 4

4 Discussion and conclusion

The results show that retrieving an update does not guarantee the model will use it: obsolete competing evidence can be more harmful than unrelated evidence. The conclusions are bounded by the models, benchmark, and retrieval settings studied.

Answers to the research questions

  • Larger budgets lower accuracy even when the latest statement is present; losses concentrate on conflict questions, and replacing unrelated evidence with an obsolete competitor harms both models.Every harmful flip in the equal-size intervention selected the stale value.Source: §4 · p. 4

Evaluation and system implications

  • The study recommends distinguishing latest-evidence retrieval, obsolete-evidence retrieval, and version selection when evaluating memory systems.It motivates retiring, down-weighting, or marking superseded facts inactive while preserving provenance and validity information.Source: §4 · p. 4

Model variation and mechanism

  • Qwen3’s smaller drop suggests model choice may affect robustness, but familiar-world-knowledge overlap means the results do not isolate temporal obsolescence as the cause.The intervention also does not establish whether attention, prior strength, or decoding dynamics explain the behavior.Source: §4 · p. 4

Scope and broader impact

  • The study covers two Qwen models, deterministic decoding, one synthetic single-hop benchmark, and lexical retrieval, so it does not establish the same behavior for dense retrieval or multi-hop systems.The questions share one benchmark history, limiting population-level statistical interpretation.Source: §4 · p. 4

A.1 Dataset and chronological semantics

The methods use chronological synthetic facts, BM25 evidence selection, frozen prompting, and paired exact-match evaluation. Support audits check whether the latest statement was retrieved, while remaining diagnostic rather than deployable retrieval procedures.

A.1 Dataset and chronological semantics

  • The 6K source has 455 statements and 100 questions, while the 32K source has 2,310 statements and 100 questions; higher-numbered conflicting statements replace lower-numbered ones.Because facts are synthetic, ordinary world knowledge is not the evaluation target.Source: §A.1 · p. 7

A.2 Models, Ollama, and decoding

  • Final controlled runs use temperature 0, seed 42, num_ctx=2048, and num_predict=64; Qwen3 additionally sets think=false.The experiments evaluate Qwen2.5-7B-Instruct and Qwen3-14B.Source: §A.2 · p. 7

A.3 Frozen final prompt

  • The frozen prompt requires using only supplied numbered memories, treating higher-numbered conflicts as newer replacements, and returning the shortest answer phrase without explanation.The prompt text is split across the supplied passages.Source: §A.3 · p. 7

A.4 Retrieval and support audits

  • BM25 ranks individual statements, adds them until the token budget is reached, then restores chronological order; gold-assisted audits check for the intended latest statement after retrieval.The audits diagnose support presence rather than provide deployable retrieval metrics; at 32K, direct inspection found the benchmark-gold statement in all 100 contexts at both budgets.Source: §A.4 · p. 7

A.5 Scoring and paired tests

  • The official metric is normalized substring exact match, with predictions also categorized as latest, stale, unknown, or other; paired conditions use exact two-sided McNemar tests.The p-values describe this benchmark instance, not a population of independent histories.Source: §A.5 · p. 7

B Complete experiment inventory

The experiment sequence moves from preliminary diagnostics toward tests of obsolete-evidence effects and robustness. Table 3 is identified as the complete experiment inventory.

B Complete experiment inventory

  • Table 3 is described as a complete experiment inventory, with numbering following the saved result files.The accompanying overview identifies exploratory, diagnostic, final, replication, and excluded runs.Source: Table 3 · p. 8, B Complete experiment inventory · p. 8
  • Preliminary baselines suggested memory helped but remained limited, while increasing retrieval budget showed a degradation trend.These were diagnostic steps in the experimental sequence.Source: B Complete experiment inventory · p. 8
  • Chain reconstruction localized the budget trend to conflict questions, and an equal-size intervention tested the effect of the obsolete competing fact.Qwen3 and 32K runs tested robustness, while manual audits checked semantic and chain-label edge cases.Source: B Complete experiment inventory · p. 8

C Preliminary baselines and oracle diagnostics

The preliminary results show a benefit from full-history prompting over question-only prompting, alongside evidence that answer errors can occur despite updated facts being available. Oracle diagnostics provide additional, non-final evidence about model use of supplied statements.

C Preliminary baselines and oracle diagnostics

  • The full 6K history improves over question-only prompting by 18 percentage points, but this comparison should not be interpreted as a causal effect of memory.In paired outcomes, 18 questions were correct only with full history, none only without memory, and 63 were wrong under both conditions.Source: C Preliminary baselines and oracle… · p. 9
  • The first-five inspection found an incorrect answer despite the updated value appearing in the supplied history, prompting relation-specific chain reconstruction.Table 5 is identified as a full-context evidence audit; its “Gold shown” count is literal occurrence, not relation-specific support.Source: Table 5 · p. 9, C Preliminary baselines and oracle… · p. 9
  • In the three-question pilot, all cases were answered correctly with either the isolated latest statement or the complete chronological chain; a later gold-assisted diagnostic reached 97%.The 97% diagnostic used gold-selected evidence and a different prompt, so its absolute accuracy is not directly comparable with final controlled conditions.Source: Table 6 · p. 9, C Preliminary baselines and oracle… · p. 9

D Complete 6K evidence-budget analysis

On the 6K history, correctness varies across BM25 evidence budgets, and the endpoint comparison favors the smaller budget. Failures are concentrated among questions with conflicting versions, even though both versions are already retrieved.

D Complete 6K evidence-budget analysis

E Update chains and conflict localization

  • The 6K chain audit found one older conflicting value for 74 questions and no previous conflict for 26.The audit used gold-assisted post-hoc matching and is diagnostic rather than deployable.Source: E Update chains and conflict… · p. 11
  • Conflict questions lost 10 net correct answers from 64 to 512 tokens, compared with three net losses among no-conflict questions.Both the obsolete and latest target statements were present in all 74 conflict contexts at every budget, so larger budgets did not newly introduce the competitor.Source: E Update chains and conflict… · p. 11, Table 11 · p. 11, Table 12 · p. 11

F Exploratory abstaining-prompt experiment

The exploratory abstaining-prompt experiment showed a strong equal-size condition difference, but frequent UNKNOWN responses limited its value as evidence about stale selection. The subsequent pilot and frozen-prompt intervention addressed that concern.

F Exploratory abstaining-prompt experiment

  • The earlier prompt encouraged UNKNOWN, and its high abstention rate kept the exploratory outputs from serving as principal evidence.The response “goalie” for gold “goaltender” was classified as other by substring matching despite being a semantic synonym.Source: F Exploratory abstaining-prompt… · pp. 11–12
  • 31 questions were correct only with latest-plus-unrelated evidence, versus one only with old-plus-latest evidence (p = 1.54 × 10−8).UNKNOWN responses occurred on 34 questions in the first condition and 38 in the second, obscuring stale selection and motivating a non-abstaining frozen prompt.Source: Table 13 · p. 11, F Exploratory abstaining-prompt… · p. 12, Table 14 · p. 12

G Frozen prompt pilot and final cross-model intervention

H Complete 32K replication

In the 32K replication, BM25-64 had higher official aggregate accuracy than BM25-512, although the latest statement was found in every context at both budgets. A manual audit also identified a substring-scoring false positive and sampled principal flips that selected obsolete values.

H Complete 32K replication

  • The 32K run used 100 questions at each budget, with two chain-label cases flagged as ambiguous after manual review.The automated audit had initially labeled 63 questions conflict and 37 no-conflict.Source: H Complete 32K replication · p. 14
  • 69/100 (69%) for BM25-64 versus 58/100 (58%) for BM25-512.Direct inspection found the statement supporting the benchmark-gold answer in 100/100 contexts under each budget.Source: H Complete 32K replication · p. 14, Table 18 · p. 14
  • A metric audit found one false positive: “rugby union” counted correct for gold “rugby” under both budgets.Reclassifying it as stale changes marginal accuracies to 68/100 and 57/100, without changing paired discordance.Source: H Complete 32K replication · p. 15

I Manual audit

  • All 13 sampled principal flips involved direct selection of an explicitly stated obsolete value.The qualitative audit does not replace the official substring metric; the two chain ambiguities were excluded from the clean no-conflict subgroup but not from official aggregate scores.Source: I Manual audit · pp. 15–16

I.1 Additional metric and mixed-output checks

The section documents scoring and mixed-answer checks, exclusions, result checks, limitations, potential harms from stale sensitive or operational memories, and a compute-resource disclosure gap. It also records that the paper’s core method development does not involve LLMs as important, original, or non-standard components.

I.1 Additional metric and mixed-output checks

  • Treating two mixed Qwen2.5 responses as non-latest yields 73/74 (98.6%) in A and 32/74 (43.2%) in B, with 41 A-only versus none B-only and unchanged exact McNemar p-value.One of the 41 discordant Condition B responses is mixed rather than a pure stale answer; primary tables retain the official benchmark scores.Source: §I.1 · p. 16

J.1 Excluded and diagnostic runs

  • The initial Qwen3 run is excluded because thinking consumed its 64-token generation allowance, and the attempted 32K full-history run exceeded its context limits; only corrected BM25-64 and BM25-512 runs inform reported 32K results.Gold-assisted isolated/support experiments are diagnostics, while the abstaining-prompt experiment is reported only as prompt sensitivity, not combined with final non-abstaining results.Source: §J.1 · p. 16

J.2 Result verification

  • Cross-checks of canonical raw outputs against summaries and excluded-run summaries against their configurations and rationales found no inconsistencies.The archive omits private server addresses and credentials.Source: §J.2 · p. 16

NeurIPS Paper Checklist

  • The paper reports that the serving endpoint’s exact accelerator and quantization metadata were unavailable, leaving its compute-resource disclosure incomplete.Per-stage call counts and client-side timings are reported in the appendix and archived raw JSONL, respectively.Source: NeurIPS Paper Checklist · p. 20
  • The checklist notes that stale sensitive or operational memories can cause outdated recommendations or disclosures, while version-aware retirement can mitigate this risk and preserve provenance.Source: NeurIPS Paper Checklist · p. 21

Related papers

Loading 2610.36308v1…