Source-linked AI summary
Supersede: Diagnosing and Training the Memory-Update Gap in LLM Agents
Vedant Patel
TL;DR
Long-horizon agents must use current facts after updates, yet bounded self-maintained memory leaves a supersession gap. The paper measures that gap and introduces Supersede, an environment that rewards current-value answers and supports training. On real held-out conversations, GRPO fine-tuning nearly doubles a small model’s accuracy, though the training evidence is from a single run.
Problem
It is unclear whether agents can reliably use current fact values after updates when operating over long interactions with bounded, self-maintained memory.
Source: §1 · p. 1Method
Results
9.0%→16.7%: GRPO fine-tuning Qwen2.5-3B nearly doubles held-out supersession accuracy on real, unseen conversations in a single run.
Source: §1 · p. 2Takeaways & Limitations
Training on Supersede moves held-out accuracy monotonically, while neither larger models nor larger memories close the gap.
Source: §1 · p. 2, §6 · p. 8The training result uses one small model and one run, and its 16.7% accuracy remains below the frontier full-context ceiling.
Source: §7 · p. 8
Abstract
from arXiv · showhide
Large language model (LLM) agents operate over long, multi-session interactions in which facts change: a user moves, a price updates, a plan is revised. Acting correctly requires using the current value of a fact and discarding values that have been superseded. We isolate this ability on real conversational data and show that it is a distinct, unsolved failure. On the knowledge-update subset of LongMemEval, replacing an agent's full context with a bounded, self-maintained memory drops accuracy from 92% to 77% even on a frontier model (gpt-5.4), a gap that is statistically significant (paired McNemar p<0.005) and persists across model scale while full-context accuracy saturates near 92%. The bottleneck is therefore memory maintenance, not comprehension, and is not closed by a stronger model. We then ask whether this is merely an undersized memory, and find it is not: as the conversation grows 24x, accuracy falls further (from 68% to 28%), and granting the agent proportionally more memory yields no detectable recovery (28% to 28%, n=25). The failure scales with the length of the conversation, not the compression ratio. We release Supersede, an open reinforcement-learning environment (on the verifiers / prime-rl stack) that turns this measurement into a training signal: agents are rewarded for answering from the current value and penalized for stale ones. Finally, we close the loop and show the gap is trainable: GRPO fine-tuning a small open model (Qwen2.5-3B) on this environment nearly doubles its held-out supersession accuracy on real, unseen conversations (9.0% to 16.7%, a single run), along a monotonic checkpoint curve indicating the learned policy, not the harness, carries the gain. To our knowledge this is the first trainable environment whose reward targets temporal fact-currency, and the first evidence the supersession gap can be trained down, not only measured.
1 Introduction
Supersession—using current facts after updates or retractions—is a distinct challenge for long-horizon agents that must maintain bounded memory. The paper measures the gap, rules out larger models and memories as fixes, and reports that training can improve held-out performance.
1 Introduction
- The paper frames supersession as maintaining current fact values after updates or retractions, and calls the bounded-memory accuracy loss the supersession gap.Its investigation asks whether the failure exists, whether model or memory scaling fixes it, and whether training does.Source: §1 · pp. 1–2
- 92%→77%: gpt-5.4 has significantly lower knowledge-update accuracy with bounded memory than with full context (p = 0.0033).The full-context advantage persists as model scale increases.Source: §1 · p. 2
- 68%→28%: accuracy falls as conversations grow 24×, while proportionally more memory yields no detectable recovery (28%→28%, n = 25).The failure tracks conversation length rather than compression ratio.Source: §1 · p. 2
- Supersede is a verifiers/prime-rl environment that makes supersession correctness a training reward rather than only an evaluation target.It reframes the forgetting-aware metric of Uddin et al. (2026) as a learning signal.Source: §1 · p. 2
- 9.0%→16.7%: GRPO fine-tuning Qwen2.5-3B nearly doubles held-out supersession accuracy on real, unseen conversations in a single run.Accuracy rises monotonically as training progresses.Source: §1 · p. 2
2 Related Work
Prior work measures long-term memory and explores memory-management systems, but the cited benchmarks are evaluation-only and existing RL rewards do not directly target whether a changed fact remains current. Supersede distinguishes itself through that reward target and a bounded-memory setting without re-feeding raw history.
2 Related Work
- LongMemEval (Wu et al., 2025) explicitly tests knowledge updates; the surveyed long-term memory benchmarks and metrics are evaluation-only.Other cited work includes LoCoMo (Maharana et al., 2024), MemoryArena (He et al., 2026), MemoryAgentBench (Hu et al., 2025), MemBench (Tan et al., 2025), and FAMA (Uddin et al., 2026).Source: §2 · p. 2
- The environment, trained model, and dataset are publicly available, and the environment is also published on the Prime Intellect Environments Hub.Its infrastructure uses verifiers and the Prime Intellect Environments Hub and is compatible with the emerging OpenEnv standard.Source: §2 · pp. 2–3
- Unlike prior RL-for-memory methods, Supersede forbids re-feeding raw history and rewards whether the agent keeps the current value of a changed fact.MemAgent (Yu et al., 2025) rewards final-answer correctness, LongRLVR (Chen et al., 2026) evidence relevance, and MemPO memory task-sufficiency.Source: §2 · p. 3
- Production memory systems include OpenAI’s “Dreaming”, Anthropic’s Claude memory, and Google’s Gemini personalization, but are closed systems.The passage notes no released benchmark, methodology, or trainable signal for these systems.Source: §2 · p. 3
3 The Supersede Environment
Supersede presents sessions sequentially, requiring an agent to maintain bounded notes and answer a changed-fact query from memory alone. Its reward programmatically scores current-value answers and can penalize stale ones when superseded values are known.
3 The Supersede Environment
- The agent processes sessions one at a time, rewrites a notes memory capped at B characters, and answers the final current-value query from memory alone.Raw sessions are never re-fed.Source: §3 · p. 3
- The primary reward is rcur = 1[current value conveyed], scored by a programmatic matcher; when superseded values are known, rstale penalizes asserting them.The reported runs optimize rcur alone; the stale penalty is used only in an environment ablation.Source: §3 · p. 3
- Tasks come from LongMemEval’s real knowledge-update subset and a procedural synthetic-timeline generator, in a verifiers MultiTurnEnv.Prompt assembly enforces no re-feeding, and rollouts end when the answer is produced.Source: §3 · p. 4
4 Experimental Setup
The experiments compare full context with session-by-session bounded memory on LongMemEval knowledge-update questions, then separately test transfer from synthetic training to real held-out questions. The setup varies history length, memory budget, models, and grading procedures.
4 Experimental Setup
- Synthetic timelines with explicit updates score 100% for frontier models reading full history, so the study uses real conversations with implicit, paraphrased updates.The generator remains a control and training curriculum.Source: §4 · p. 4
- The oracle evaluation uses all 78 LongMemEval knowledge-update questions, while the longer s split has about 48 sessions and 122k tokens per question.The oracle split is graded by an LLM judge; the longer split uses a programmatic matcher held constant across conditions.Source: §4 · p. 4
- FULL-CONTEXT includes every session; BOUNDED-MEMORY rewrites a 300-character notes field session by session without re-feeding raw sessions.The scale study also tests a constant-ratio budget of 150 characters per session.Source: §4 · p. 4
- Training uses Qwen2.5-3B-Instruct with GRPO on 2,000 synthetic episodes, then evaluates transfer on real LongMemEval oracle questions never seen in training.Base and trained models use the same programmatic matcher, and held-out evaluations use deterministic greedy decoding.Source: §4 · pp. 4–5
- Figure 2 compares full-context and bounded-memory knowledge-update accuracy across three models; full-context reaches 92% while bounded-memory does not catch up.Its caption identifies n = 78 oracle questions and the LongMemEval judge protocol.Source: Figure 2 · p. 5
5 Results
GRPO training on Supersede nearly doubles held-out supersession accuracy, while larger models and proportionally larger memories do not close the bounded-memory gap. The results link failures to memory maintenance across longer conversations and show that the training curriculum itself saturates.
5.1 Bounded memory degrades supersession
- 92% to 77%: on gpt-5.4, bounded memory significantly lowers accuracy relative to full context, while full-context accuracy saturates across the model family.For gpt-5.4, paired McNemar p = 0.0033; the authors identify memory maintenance, rather than comprehension, as the bottleneck.Source: §5.1 · p. 5
- Errors include relevant facts being compressed away or not overwritten, leaving agents unable to retrieve updated answers from memory.Examples include a missing restaurant count and a missing updated location for Rachel.Source: §5.1 · pp. 5–6
5.2 It is scale, not memory size
- 68% to 28%: growing conversation length about 24× lowers accuracy at fixed memory, and 24× more memory leaves accuracy at 28%.The study used gpt-4.1-mini and n = 25; proportional memory produced no detectable recovery.Source: Figure 3 · p. 6, §5.2 · p. 6, Table 2 · p. 6
5.3 Training closes the gap
- 9.0% to 16.7%: GRPO training raises held-out LongMemEval oracle accuracy for Qwen2.5-3B, with a monotonic gain across checkpoints.The rise begins as procedural reward saturates, and accuracy is still increasing at the final checkpoint; this was a single training run.Source: Figure 4 · p. 7, §5.3 · pp. 7–8, Table 3 · p. 7
5.2 It is scale, not memory size
- 57.7%: the released environment's programmatic reward on all 78 questions is consistent with the 63% measured by the LLM judge.The matcher is stricter, and every rollout terminates cleanly.Source: §5.2 · p. 7
5.3 Training closes the gap
- 0.66 to 0.97: GRPO reward on procedural episodes rises before training self-terminates at step 175 when group-relative advantage vanishes.The procedural distribution is solved, limiting what this curriculum can teach.Source: §5.3 · p. 7
6 Discussion
The discussion frames supersession as a memory-policy problem: neither increasing model size nor allocating more memory closes the gap, while training a bounded-memory policy improves held-out accuracy.
6 Discussion
- Model size and memory size do not close the gap, but GRPO training on Supersede nearly doubles held-out accuracy with the same bounded-memory agent.The lift is partial; the discussion argues for optimizing a policy to preserve currency rather than waiting for a larger model.Source: §6 · p. 8
7 Limitations
The training result is an initial proof of mechanism, limited by a single small-model run, a saturating procedural curriculum, and caveats in the diagnosis and grading.
7 Limitations
- The training result is a single Qwen2.5-3B run, and its 16.7% accuracy remains far below the frontier full-context ceiling.The reported absolute gain is +7.7 points.Source: §7 · p. 8
- Procedural episodes are solved before training ends, so this curriculum cannot teach harder supersession than it contains.The authors identify longer, more implicit episodes that do not saturate as the next lever.Source: §7 · p. 8
- The scale study uses n = 25 rather than n = 78, and adds distractor sessions rather than more updates to the tracked fact.The constant-ratio null is therefore measured on a smaller, noisier sample, and conversation length is not identical to update frequency.Source: §7 · p. 8
- The frontier gap is judge-graded while the training comparison uses a lenient matcher, making their absolute accuracies not directly comparable.The base-to-trained delta uses the same matcher, but the single-run gain lacks formally established statistical significance at n = 78.Source: §7 · p. 8
8 Conclusion
The paper concludes that bounded-memory agents fail at supersession through maintenance rather than reading, and that training on Supersede partially closes this gap.
8 Conclusion
- Supersession failures reflect difficulty maintaining updates under bounded memory, not difficulty reading them; training on Supersede partially closes the gap.The conclusion reports nearly doubled held-out accuracy for a small open model, with a monotonic learning curve.Source: §8 · p. 9
A Reward matcher
The reward matcher compares an answer against the current gold value and known stale values using normalization, variants, substring matching, and a fallback.
A Reward matcher
- A programmatic, judge-free matcher compares model answer a with the gold current value g for reward scoring.The matcher can also be run against known superseded values to penalize stale answers.Source: A Reward matcher · p. 10
- Normalization lowercases strings, replaces punctuation with spaces, and collapses whitespace; gold connectives split g into acceptable variants of length at least two.For example, “25 minutes and 50 seconds” accepts either component as a variant.Source: A Reward matcher · p. 10
- An answer matches when a normalized gold variant occurs as a substring, or when token overlap with a variant exceeds 0.6.The token-overlap rule is a fallback when no substring variant matches.Source: A Reward matcher · p. 10
- A current-value match yields r_cur = 1[match], while a match to a known stale value triggers the stale-answer penalty r_stale.The matcher is lenient on surface form but strict about which value is reported.Source: A Reward matcher · p. 10
B Training details
Training uses procedural bounded-memory episodes and GRPO; the run stops when sampled rollouts all succeed and the group-relative learning signal vanishes.
B Training details
- Training episodes contain 6–8 sessions, with one introduced fact later superseded and its update buried among distractor sessions.Evaluation uses the untouched real LongMemEval oracle set.Source: B Training details · pp. 10–11
- At step 174, ten consecutive zero-advantage batches ended training because every sampled rollout succeeded and the GRPO gradient vanished.This run therefore self-terminated rather than continuing through a fixed training schedule.Source: B Training details · p. 11
- Table 4 lists the full GRPO training configuration.The supplied passage identifies the table's scope but does not give its configuration values.Source: B Training details · p. 10, Table 4 · p. 11
C Example episodes
The procedurally generated episodes test whether an agent’s bounded memory preserves the latest fact across sessions and excludes superseded values. One example has a single update; another has two.
C Example episodes
- The episodes reward an answer conveying the current value rather than a superseded one, while the agent sees sessions sequentially and maintains bounded notes without re-fed raw sessions.The reward is 1 only for conveying the current value.Source: C Example episodes · p. 11
- In the seven-session city episode, Atlanta supersedes Chicago, and the query asks which city the user currently lives in.The intervening sessions contain distractors.Source: C Example episodes · p. 11
- In the six-session car episode, Lexus supersedes both Kia and Mazda, and the query asks what car the user currently drives.The episodes include two updates and intervening distractors.Source: C Example episodes · p. 11