Source-linked AI summary
Temporal Validity on Real Software Histories: Eliminating Stale-Fact Errors in Code-Assistant Memory over GitHub Fixes
Neeraj Yadav
TL;DR
RAG lacks temporal validity for changing code facts, motivating a deterministic supersession memory. The paper evaluates that mechanism on clean atomic transitions from real GitHub histories and finds higher accuracy and near-elimination of stale answers, within a limited extraction scope.
Problem
RAG retrieves old and new versions of changing code facts without representing which value is current.
Method
The paper extracts marker-free atomic transitions from real SWE-bench histories and evaluates deterministic triple-based supersession memory on them.
Results
0.91 accuracy versus RAG’s 0.57–0.59, while forced stale-fact answers fall from 36–38% with RAG to approximately zero with MemStrata.
Takeaways & Limitations
On clean atomic code mutations, deterministic supersession generalizes the synthetic result to real software histories and preserves the stated moat after a product bug fix.
Takeaways & Limitations
The evaluation covers approximately 18% of real fixes that qualify as clean atomic transitions; broader extraction coverage is deferred.
Abstract
from arXiv · showhide
Retrieval-augmented generation (RAG) has no model of time: when a fact changes across a coding session - a function is renamed, an endpoint moves, a dependency is bumped - RAG retrieves both the old and new value with near-identical similarity and cannot tell which is current, so it serves the superseded value. Paper 1 showed, on synthetic single-value benchmarks, that a deterministic (subject, relation, object) supersession memory eliminates this failure. Here we validate it end-to-end on real software history. From 707 real GitHub issues (SWE-bench Lite + Verified) we extract 130 clean atomic state transitions, a fix that changes one identifiable value from a pre-fix to a post-fix form, and render each marker-free (the stale and current statements differ only in the value). On this set, MemStrata reaches 0.91 answer accuracy versus RAG's 0.57-0.59; and, the structural result, when forced to answer RAG serves the superseded value 36-38% of the time (an LLM reranker does not help) while MemStrata drives this to ~0, at RAG retrieval latency (~2.1 s vs ~18 s for the reranker). We are explicit about scope: only ~18% of real fixes are clean atomic transitions; Paper 2 isolates the memory mechanism on that class, and extraction coverage of the remaining fixes is the orthogonal problem we defer to follow-on work. A real product bug surfaced and was fixed during the study (a case/punctuation-insensitive value comparison), with the moat property (deterministic-supersession accuracy on clean code mutations) preserved and verified.
1 Introduction
RAG lacks a model of fact currency, so it can retrieve both superseded and current code values without knowing which is current. This paper addresses that gap with a real-history benchmark and reports a substantial MemStrata advantage.
- RAG stores changed statements near each other and cannot reliably distinguish current values from superseded ones.
- Paper 1’s synthetic finding motivates testing deterministic supersession on real GitHub buggy→fixed histories.
- 130 clean atomic state transitions were extracted from 707 real GitHub issues and rendered marker-free.Each transition contains one verified pre-fix value and one post-fix value.
- MemStrata reaches 0.91 accuracy versus RAG’s 0.57–0.59 on the real-history benchmark.
- The study separates the memory mechanism from extraction coverage and reports both abstention-allowed and forced-answer evaluations.A product bug was also disclosed and fixed while preserving the method’s stated moat.
2 Related Work
Prior memory and RAG systems emphasize recall but generally lack fact currency, while this work repurposes SWE-bench histories to evaluate whether changed facts remain current. It extends a bi-temporal memory idea from synthetic benchmarks to real software histories.
- Persistent-memory and graph-RAG systems emphasize recall or long-context handling without introducing fact currency.
- SWE-bench’s real buggy→fixed issue–patch pairs are repurposed to test memory currency rather than patch-writing ability.
- Paper 1 adapts established bi-temporal database modeling to LLM memory, and this paper validates that architecture on real data.
3 Method
MemStrata applies deterministic supersession to value-bearing triples in a bi-temporal ledger. Scenarios are mined and retained only when real fixes yield clean, marker-free atomic transitions that the mechanism can engage.
- MemStrata supersedes an active triple when its subject–relation key matches but its object differs.The old validity interval closes, the new one opens, and retrieval exposes only currently valid rows.
- Each scenario begins as one atomic pre-fix/post-fix value change extracted from a SWE-bench problem statement and unified diff.
- Scenarios are rejected unless the values differ, remain short and atomic, and contain no recency cue.
- A scenario is retained only when production extraction assigns both state sentences the same key and the two corresponding objects.This makes selection tied to the defined supersession mechanism rather than manual preference.
- The marker-free construction makes order the only currency signal, yielding 130 clean scenarios from 707 records, or 18.4%.
4 Experimental Setup
The experiment compares four memory conditions across paired allowed and forced regimes on identical cached scenarios. It prioritizes answer-level stale-fact error, with accuracy and other diagnostics as secondary measures.
- All runs use local deterministic inference with fixed seeds and designated answer, judge, and embedding models.
- The dataset contains 707 deduplicated, sha256-pinned records from SWE-bench Lite and Verified.
- Four conditions compare no memory, naive RAG, RAG with an LLM reranker, and temporal_v6.All conditions ingest the same turns and answer the same questions.
- Allowed and forced regimes run on the same cached 130 scenarios, with forced answering exposing stale commitments that abstention can hide.
- Answer-level stale-fact-error is primary, followed by accuracy, fabrication, compression, and supersession-correctness diagnostics.Ledger-level stale_survivors is explicitly treated as non-headline because cross-scenario collisions inflate it.
- The paired allowed rerun reproduces naive_rag 0.569, advanced_rag 0.585, and temporal_v6 0.908 exactly.
5 Results
On 130 paired real GitHub scenarios, MemStrata sharply reduces stale answers versus RAG and preserves this advantage at comparable retrieval latency.
- 0.985 supersession-correctness and a 1.023 bounded-growth ratio were reported for the temporal mechanism.
- 36.1% of forced RAG answers served the superseded value, compared with approximately 0 for MemStrata.RAG’s allowed stale-fact-error was 26.2%; MemStrata’s forced range was 0.000–0.015 across runs.
- 0.91/0.99 allowed/forced accuracy was achieved by MemStrata, versus 0.57–0.62 for RAG.
- Approximately 48% bounded-growth compression was obtained at RAG retrieval latency.
6 Discussion
The discussion separates temporal-memory correctness from extraction coverage, reports a reranking negative result, and documents a fixed product bug without losing the safety property.
- Mechanism versus coverage: Approximately 18% of real fixes were evaluated because the study conditions temporal correctness on clean atomic extraction.Extraction coverage for multi-value, logic, and behavior fixes is deferred as orthogonal follow-on work.
- Mechanism versus coverage: RAG failed on the same selected subset, with 0.57 accuracy and 36% stale answers, so the subset defines the temporal-memory regime rather than a winning-only regime.
- Product bug: 1.000 accuracy and 0.000 stale-fact-error on code_mutation were preserved after fixing the case/punctuation-insensitive comparison bug.The full unit-test suite also stayed green.
- Reranking: 37.7% stale answers from advanced_rag when forced showed that learned reranking tracked naive RAG and did not add a temporal signal.
7 Limitations •
The paper’s limitations are concentrated in dataset coverage, sample size, model setting, and small answer-model variance rather than the deterministic pipeline.
- Approximately 18% of real fixes were clean atomic transitions; extending beyond them requires extraction-robustness work.
- 130 real scenarios constitute a focused mechanism result, not a leaderboard ranking.Scaling to more records is identified as future work.
- A single 7B local model on consumer hardware limits the reported model setting.Larger or cloud models may shift absolute baselines, though the passage states the structural gap may remain.
- Approximately one question of answer-model run-to-run variance remained, while the deterministic pipeline was unaffected.
8 Conclusion
On clean atomic transitions from real GitHub histories, deterministic supersession memory generalizes Paper 1’s synthetic result: it preserves current values while RAG frequently returns stale ones.
- 0.91–0.99 accuracy for MemStrata exceeds RAG’s 0.57–0.62 on clean atomic transitions.
- 36–38% stale-fact errors from forced RAG fall to ≈0 with deterministic supersession memory at RAG latency.
- The memory mechanism generalizes from synthetic benchmarks to real GitHub buggy→fixed histories, while arbitrary-fix extraction coverage remains future work.
Reproducibility Statement
The evaluation pipeline is deterministic, locally reproducible, and based on pinned data and cached paired scenarios.
- The pipeline uses temperature 0, fixed seeds, no network, and sha256-pinned SWE-bench data.
- Cached scenarios pair allowed and forced evaluation regimes under identical inputs.
- Gate-fix tests, a moat guardrail, and an end-user stale-correction demonstration support reproducibility.
A Scenario construction and examples
Scenarios represent verified, marker-free atomic changes mined from real GitHub fixes, with examples covering identifiers, paths, and dependency versions; a comparison bug was fixed and guarded.
- Scenario definition: 130 clean scenarios are verified atomic transitions rendered marker-free, so state-A and state-B differ only in the changed value.
- Examples: Examples change a login-handler name, an API base path, and a FastAPI version while asking for the state-B value.
- Selection criterion: A scenario is admitted only when production extraction keys both turns identically and assigns the two values as objects.
- Extraction: The patch reader emits one atomic old→new value change or declines multi-value and non-atomic fixes.
- Bug fix and guardrail: A case- and punctuation-insensitive comparison previously dropped some supersessions; strict comparison now preserves them without changing code_mutation accuracy 1.000 or stale-fact-error 0.000.