Source-linked AI summary
Temporal Validity in Retrieval Memory: Eliminating Stale-Fact Errors for AI Agents over Evolving Knowledge
Neeraj Yadav
TL;DR
RAG retrieves stale and current facts without representing time, leaving agents to abstain or risk serving superseded values. MemStrata uses deterministic supersession in a bi-temporal ledger while retaining facts like RAG; it reaches 0.95–1.00 accuracy on evolving benchmarks, while RAG serves superseded values 15–40% of the time when required to answer.
Problem
RAG has no representation of time, so it retrieves old and current versions with near-identical embeddings and cannot reliably identify which is current.
Source: §1 · p. 2Method
MemStrata stores facts like RAG and uses a deterministic (subject, relation, object) supersession rule to retire contradicted values in a bi-temporal ledger.
Source: §3 · p. 4, §4.3 · p. 5Results
Takeaways & Limitations
For evolving knowledge, the paper identifies currency rather than recall as the binding memory failure and recommends marker-free evaluation of memory under evolution.
Source: §8 · pp. 8–9The evolving benchmarks are structured single-value templates, and extraction quality gates performance on messier natural-language contradictions.
Source: §7 · p. 8
Abstract
from arXiv · showhide
Retrieval-augmented generation (RAG) gives agents access to accumulated knowledge, but has no model of time. When a fact changes (e.g., a function is renamed or API restructured), RAG retrieves both the stale and current value with near-identical embedding similarity. The agent then either abstains or serves the superseded fact. We show this is a structural problem: on a calibrated dataset, cosine similarity distinguishes a contradicted fact from a duplicated one with AUROC 0.59 (near chance), as contradictions are often more embedding-similar to the original than rephrased duplicates. We present MemStrata, a retrieval memory maintaining temporal validity. It stores facts like RAG, preserving static recall, but when a fact's value is contradicted, a deterministic (subject, relation, object) supersession rule retires the stale value in a bi-temporal ledger - with no similarity threshold and no LLM call. Across six benchmarks run locally with a 7B model, MemStrata ties RAG on static knowledge and reaches 0.95-1.00 accuracy on evolving knowledge (where RAG reaches 0.20-0.47). The central result is the stale-fact-error rate: when required to answer, RAG serves superseded values 15-40% of the time; MemStrata drives this to ~0%, a failure class RAG cannot avoid. MemStrata achieves this at retrieval latency (~2.1s) versus ~16-18s for LLM-reranking baselines. We release the harness, datasets, and a marker-free evaluation protocol for memory under knowledge evolution.
1 Introduction
RAG retrieves old and current facts without representing which is valid, leaving agents to abstain or risk serving stale values. MemStrata uses deterministic supersession to retain static recall while addressing that currency problem.
1 Introduction
- On 98 labeled pairs, cosine AUROC is 0.59 for separating contradictions from duplicates, with maximum achievable precision of 0.67.Contradictions can be more embedding-similar to the original than genuine rephrasings because value changes are minimal edits.Source: §1 · p. 2
- MemStrata applies a deterministic (subject, relation, object) supersession rule and retires contradicted values in a bi-temporal ledger without a similarity threshold or LLM call.It stores facts like RAG to preserve static recall.Source: §1 · p. 2
- RAG serves superseded values 15–40% of the time across four evolving benchmarks, while MemStrata drives stale-fact errors to ∼0%.When required to answer, RAG has no mechanism to choose between retrieved old and new values.Source: §1 · p. 2
- The marker-free protocol makes stale and current fact versions textually identical except for the changed value, isolating temporal mechanisms as the currency signal.A contaminating textual marker can silently inflate baseline scores.Source: §1 · pp. 2–3
- The evolving benchmarks are structured single-value templates, and extraction quality—not supersession—is the gating factor for messier natural-language contradictions.The authors identify this as a limitation bounding their claim.Source: §1 · p. 3
2 Related Work
Prior memory systems emphasize conversational recall, graph RAG enriches relations without representing currency, and bi-temporal ledgers have established database roots. MemStrata applies deterministic supersession to agent memory and evaluates stale-fact resistance under knowledge evolution.
Memory for LLM agents
- Persistent-memory systems target conversational settings and emphasize recall depth, while MemStrata targets stale-fact resistance under knowledge evolution.Examples include long-term memory pipelines [Mem0; Chhikara et al., 2025], OS-style hierarchies [MemGPT/Letta; Packer et al., 2023], and reflective memory.Source: §2 · p. 3
Graph and hypergraph RAG
- GraphRAG [Edge et al., 2024] and related graph systems enrich relational representations but still retrieve by similarity and introduce no notion of fact currency.MemStrata is orthogonal: it adds temporal validity and evaluates on evolving rather than static corpora.Source: §2 · p. 3
Temporal knowledge graphs and bi-temporal data
- Bi-temporal modeling separates valid time from transaction time, and MemStrata adapts the ledger to agent memory by retiring rather than deleting facts.The paper distinguishes its contribution—the integration of deterministic extraction-time supersession and empirical demonstration—from the established ledger primitive [Snodgrass and Ahn, 1985; Snodgrass, 1999, Jensen and Snodgrass, 1999; ISO/IEC, 2011].Source: §2 · p. 3
Hallucination and verification
- Verification-augmented RAG does not address staleness because it lacks a temporal signal, and the included LLM relevance-verifier baseline costs ∼8× latency.The paper contrasts learned grounding checks with a temporal, deterministic mechanism.Source: §2 · p. 3
3 The Staleness Problem and Why Similarity Cannot Solve It
RAG can retrieve both versions of an evolving fact but has no basis for choosing the current one. Similarity cannot reliably distinguish a contradiction from a duplicate, motivating structural supersession keyed by subject and relation.
3 The Staleness Problem and Why Similarity Cannot Solve It
- A query over an evolving store retrieves both old and new values, which are near-identical in embedding space, and RAG provides no basis for choosing the current one.Retrieval ranks by similarity, so both versions can be maximally similar to the query.Source: §3 · p. 4
- A write-time update must distinguish duplicate, contradiction, and novel facts, but a similarity threshold would work only if similarity separated duplicates from contradictions.A value change can be a smaller textual edit than a rephrasing of the same fact.Source: §3 · p. 4
- Cosine similarity cannot separate contradictions from duplicates: value-flips are smaller edits than genuine rephrasings, and threshold precision reaches at most 0.67.A learned classifier over similarity features does not reliably rescue the approach; in the abstention regime, gate-judge conditions leak stale facts 25–60% of the time.Source: §3 · p. 4
- MemStrata uses structural supersession: a newer fact with the same (subject, relation) key and a different object supersedes the older value, independent of embedding similarity.The rule implements the paper’s conclusion that staleness detection must be structural.Source: §3 · p. 4
4 The MemStrata Architecture
MemStrata sits between an agent and its language model, storing extracted facts and composing token-budgeted context. Its write path preserves distinct facts while deterministically superseding changed values, and its read path filters retired rows before packing context.
4 The MemStrata Architecture
- MemStrata is a local memory layer that stores facts extracted from interaction and composes a token-budgeted context block for each query.Source: §4 · p. 4
4.1 Write path: deterministic supersession over a surprise gate
- For clean single-value triples, the write path keys on normalized (subject, relation): a different object supersedes the active assertion, while a same object reinforces it and a missing key is stored as novel.The fallback routes non-triple prose through a surprise gate with similarity and an LLM judge; it retains non-contradictory near-duplicates and drops only exact duplicates.Source: §4.1 · p. 4
4.2 Bi-temporal ledger
- Superseded facts are retired rather than deleted, with valid_from, valid_to, and superseded_by recorded for possible future as-of-time queries.Active retrieval surfaces only currently valid rows; as-of-time querying is not evaluated here.Source: §4.2 · p. 4
4.3 The “retain, then supersede” design
- The published configuration retains distinct non-contradictory facts like RAG and bounds growth by superseding contradictions, avoiding the static-recall loss of aggressive merging.The lossy ablation drops to 0.62 on project-fact QA and 0.13 on dialogue recall.Source: §4.3 · p. 5
4.4 Read path
- The read path retrieves top-k active facts by cosine, drops superseded rows, and packs each surviving assertion’s original source sentence; no LLM runs on this path.Retrieval latency is ∼2.1 s versus ∼16–18 s for LLM-reranking and LLM-verification baselines.Source: §4.4 · p. 5
4.5 Marker-free benchmark construction
- The marker-free invariant keeps stale and current versions textually identical except for the changed value, so ingestion order is the only available currency signal.Removing a marker from an earlier benchmark dropped baseline accuracy by up to 14 points, and the authors recommend the invariant for evaluating memory under evolution.Source: §4.5 · p. 5
5 Experiments
Experiments compare MemStrata with RAG and alternative retrieval conditions on static and evolving benchmarks, assessing accuracy, stale-fact errors, memory growth, and latency.
5 Experiments
- Eight conditions span six benchmarks—two static and four evolving—and are assessed with accuracy, stale-fact error, fabrication, active-fact count, compression, and latency metrics.The forced-answer supplement disables abstention on RAG conditions.Source: §5 · pp. 5–6
5.1 Cosine cannot separate contradictions from duplicates
- 0.5926 cosine AUROC separates duplicates from the rest; contradictions average 0.812 similarity versus 0.800 for duplicates, with maximum threshold precision of 0.667.The stated 0.95 precision floor for a safe automatic rule is unreachable.Source: Table 1 · p. 6, §5.1 · p. 6
5.2 Accuracy: ties RAG on static, dominates on evolving
- 0.95–1.00 versus 0.20–0.47 evolving accuracy, while temporal_v6 ties RAG on static accuracy (0.82/0.30 vs 0.86/0.30).The comparison covers the two static and four evolving benchmarks.Source: Table 2 · p. 6, §5.2 · p. 6
5.3 Stale-fact error: the structural result
- 15–40% of forced-answer contradiction questions receive superseded values under RAG, while MemStrata reaches ∼0% in both regimes.The stale-fact-error rate measures contradiction questions answered with a superseded value.Source: §5.3 · pp. 6–7, Table 3 · p. 7
5.4 State-bounded growth
- ∼48% compression on evolving facts bounds memory growth, while MemStrata retains distinct static facts with 0% compression.Evolving compression ranges from 47.5% to 50% across the four benchmarks.Source: §5.4 · p. 7
5.5 Latency
- ∼2.1 s retrieval latency for temporal_v6 matches naive RAG, versus ∼16–18 s for LLM reranking or verification conditions.The reported temporal_v6 latency holds on both static and evolving benchmarks.Source: §5.5 · p. 7
6 Discussion
The discussion frames MemStrata as preserving RAG’s static-knowledge performance while addressing stale facts in evolving knowledge, and identifies both design boundaries and evaluation concerns.
6 Discussion
- For evolving knowledge, RAG serves stale facts 15–40% of the time when forced to answer, while deterministic supersession eliminates that error at the same retrieval latency.For static knowledge, MemStrata matches RAG.Source: §6 · p. 7
- The ablations bracket the design: over-merging sacrifices static recall, while removing supersession collapses evolving accuracy from 0.99 to 0.33 and re-raises stale-fact error.The no-supersession result is statistically indistinguishable from naive RAG at 0.32.Source: §6 · pp. 7–8
- An earlier benchmark with textual staleness markers inflated baseline accuracy by up to 14 points.The paper’s marker-free protocol is intended to evaluate memory under knowledge evolution without those markers.Source: §6 · p. 8
7 Limitations
The reported boundaries are structured single-value benchmarks, ingestion order as a proxy for time, a single-judge issue, and small-scale local evaluation.
7 Limitations
- Extraction is the gating factor: structured single-value benchmarks yield ∼97% supersession, while messier natural-language contradictions yield ∼44%.The ledger and supersession rule are described as domain-independent, but the extraction layer must adapt; extraction robustness is follow-on work.Source: §7 · p. 8
- Ingestion order stands in for time; real valid_from timestamps and as-of-T retrieval are future work, although the ledger already stores validity intervals.The benchmarks use state-A followed by state-B as the currency signal.Source: §7 · p. 8
- A single 3B correctness judge sometimes marks an answer containing the stale value correct, creating a few overlaps between accuracy and stale-error.The temporal layer’s ∼0 stale-error is unaffected because no stale fact is in its context.Source: §7 · p. 8
- Evaluation uses one 7B answer model on consumer hardware and benchmarks of tens of items, which isolate mechanisms rather than rank systems on a leaderboard.Larger models, cloud inference, and real-world longitudinal data are outside the reported evaluation scope.Source: §7 · p. 8
8 Conclusion
MemStrata addresses stale retrieval by maintaining temporal validity with deterministic supersession over a bi-temporal ledger. It preserves static recall, improves evolving-benchmark accuracy, and reduces stale-fact errors, while the paper identifies time-stamped world knowledge as a next step.
8 Conclusion
- MemStrata ties RAG on static knowledge, reaches 0.95–1.00 accuracy on evolving benchmarks versus RAG’s 0.20–0.47, and reduces stale-fact errors from RAG’s 15–40% to ∼0%.The conclusion also reports retrieval latency and no language model on the read path.Source: §8 · p. 9
- A deterministic (subject, relation, object) supersession rule retires contradicted facts in a bi-temporal ledger before retrieval while storing facts like RAG.The conclusion describes this design as preserving static recall.Source: §8 · p. 9
- The paper releases a harness, datasets, and protocol, and identifies extending the architecture to time-stamped world knowledge as a natural next step.Coding is presented as the initial application, not the architecture’s sole domain.Source: §8 · p. 9
Reproducibility Statement
The reproducibility statement documents deterministic local experiments and released artifacts; the appendix result tables are reproduced from committed source reports.
Reproducibility Statement
- All experiments are deterministic, using temperature 0, fixed seeds, no network, and tests that enforce these conditions.The release includes the harness, extraction and gate prompts with content hashes, six benchmark datasets with hashes, the calibration dataset, and per-run logs.Source: Reproducibility Statement · p. 9
A Full result tables
- Tables in A.1–A.3 are reproduced verbatim from committed source reports by eval/build_appendices.py, without hand-transcription.Source: A Full result tables · p. 10
A.1 Main matrix — 8 conditions × 6 benchmarks (REPORT_PAPER1.md)
This section reports stale-fact-error and retrieval-latency outcomes and describes the evaluation materials and procedures, including marker-free benchmark construction, extraction rules, and prompts for fact classification and retrieval-relevance verification.
A.1 Main matrix — 8 conditions × 6 benchmarks (REPORT_PAPER1.md)
- Across static and evolving knowledge, temporal_v6 retrieval latency remains ∼2.1 s, while LLM-rerank/verify conditions cost ∼16–24 s throughout, demonstrating regime-independent latency.The latency matrices are split by regime and report mean and p95 metrics.Source: §A.1 · p. 12
A.2 Forced-answer supplement — no-abstention, 3 conditions × 4 evolving benchmarks (REPORT_PAPER1_forced.md)
- The forced-answer supplement evaluates stale-fact errors, conditional fabrication, and mean retrieval latency for naive_rag, advanced_rag, and temporal_v6 on four evolving benchmarks.The supplied passages identify the metrics and conditions but do not provide their values.Source: §A.2 · pp. 12–13
A.3 Cosine calibration — grader-independent (calibration/REPORT_synthetic.md)
- The grader-independent cosine calibration section includes label-wise cosine distributions and sweeps of τdup and τnovel, but the supplied passages give no results.Source: §A.3 · pp. 13–14
B Benchmark construction
- Evolving scenarios differ only in one mutated value, omit explicit staleness cues, and use ingestion order as the sole currency signal.Removing an [OUTDATED] marker lowered reranker-RAG accuracy by 14 points and gate-only accuracy by 18 points, versus 4 points for the temporal method.Source: B Benchmark construction · p. 15
- The fair_contradiction benchmark is excluded because the triple extractor keys reliably on only ∼44% of scenarios; temporal_v6 scores 0.62 there versus advanced_rag at 0.74.The authors attribute the result to extraction failures on 56% of pairs and treat the benchmark as a boundary of the method.Source: B Benchmark construction · p. 16
C Prompts and content hashes
- The single-value extractor maps changeable facts to deterministic subject-relation-object triples, while a flag-gated multi-value extractor emits one triple per changeable value.The single-value extractor targets concrete names, endpoints, configuration values, ports, versions, paths, and identifiers; non-value-bearing statements are excluded.Source: C Prompts and content hashes · pp. 17–18
- The gate judge classifies candidate facts as duplicate, merge, contradict, or novel; contradictions supersede the old fact and store the candidate as current truth.When unsure between merge and novel, the prompt says to prefer novel.Source: C Prompts and content hashes · pp. 18–19
- The v6 relevance verifier judges candidate memories as SUPPORTED, IRRELEVANT, or CONFLICTS and requires each justification to quote the candidate verbatim.It is never used on temporal_v6’s read path; correctness and fabrication judges and answer prompts are inline in eval/run_matrix.py.Source: C Prompts and content hashes · p. 19
D Ablations
The ablations test whether the temporal result depends on supersession rather than lossy compression, retention and packing refinements, or LLM verification.
D.1 retain-vs-lossy
- The lossy variant scores 0.62/0.13 on static recall, while the full method ties RAG at 0.82/0.30 and matches it on the four evolving benchmarks.The passage attributes bounded growth to retiring stale facts, not merging distinct non-contradictory facts.Source: D Ablations · p. 20
D.1b full-retention
- Removing supersession drops mean evolving accuracy from 0.99 to 0.33 and raises stale-fact error from 0.00 to 0.05–0.25.The ablation also raises mean conditional fabrication from 0.04 to 0.25, peaking at 0.56 on config_migration.Source: D Ablations · p. 20
D.2 original-text packing
- A dedicated single-factor test of original-text packing was not run, so its effect was validated jointly with the retention refinement.The reported code_mutation change from 0.80 to 1.00 reflects packing the original sentence rather than the terse triple, not an isolated packing measurement.Source: D Ablations · p. 20
D.3 LLM-verifier
- LLM relevance verification does not improve the reported static/recall tasks and raises latency from approximately 2.1 seconds to 16–18 seconds.The comparison is v6 (gate plus LLM relevance verification) versus v6_no_verify (gate only).Source: D Ablations · p. 20
D.4 +INFER
- Adding inference is neutral across benchmarks after the single-candidate tautology-guard fix.The v6+infer variant is reported as approximately equal to v6 on every benchmark.Source: D Ablations · p. 21