Source-linked AI summary
The Epistemics of Agent Memory: Measuring, and Governing, the Consolidation Decision in Long-Horizon LLM Agents
Sasank Annapureddy, Anjaneya Prasad Thamatani
TL;DR
Long-horizon agents must turn experience into durable memory, but existing systems either segment heuristically or learn consolidation on pre-given boundaries. This paper studies consolidation through learned segmentation and abstraction, oracle-based scoring, and governance. The Phase 2 policy improves task success with compression, while a two-benchmark study finds the redesigned quality score does not predict transfer accuracy.
Problem
Existing systems use heuristic episode segmentation or learn consolidation on pre-given boundaries, leaving the problem of learning boundaries from agent traces by downstream utility unaddressed.
Source: §2.1 · p. 3Method
The four-phase program studies learned episode segmentation and abstraction, builds an oracle-based consolidation benchmark, and wraps consolidation decisions in governance.
Source: §8 · p. 10Results
The Phase 2 policy yields +22.7% relative task-success improvement over the best non-learned baseline with 7.17× memory compression, while pooled Spearman ρ = −0.24 (n=12) shows CQS does not predict transfer accuracy.
Source: §3.1 · p. 4, §5.4 · p. 8Takeaways & Limitations
The paper’s durable contribution is rigorous ways to evaluate, diagnose, and govern consolidation decisions rather than four claimed solutions.
Source: §8 · p. 10CQS is validated as a consolidation-decision score, not as a predictor of downstream answer quality.
Source: §9 · p. 10
Abstract
from arXiv · showhide
Long-horizon LLM agents must convert accumulated experience into durable memory, deciding what to keep, compress, abstract into reusable skills and rules, or forget. We report a four-phase research program on this consolidation problem whose central finding is a shift in what is measured: from how much an agent remembers, to whether its consolidation decisions are any good, to whether those decisions can be trusted. Phase 1 learns episodic boundaries from agent traces by downstream utility; an honest near-miss (oracle correlation 0.691 vs a 0.70 bar) whose lasting output is a three-gate anti-leakage protocol. Phase 2 learns when to promote experience and to which abstraction level under a token budget, achieving a verified +22.7% task-success improvement with 7x compression, but exposing a degenerate-forgetting failure and a distribution-shift failure mode we name lambda-prevalence coupling. Phase 3 introduces ConsolidationBench, an oracle-by-construction benchmark that scores consolidation decisions against a known optimum on three non-circular axes; production retrieval systems retain information yet score zero on cross-level transfer. Phase 4 introduces governed consolidation: the decision wrapped in poison-resistance, reversibility, and auditability guarantees with a quality gate. Governance is statistically distinct from the quality score ($r^2 = 0.43$; partial $r = 0.27$; identical-quality policies differ threefold in governance), so the contribution survives independently of the metric's external validity. On that question we report a resolved negative: after a graded-reuse redesign removed a structural ceiling, a two-benchmark study with 2,532 real answer cells finds the quality score does not predict real transfer accuracy (pooled Spearman $\rho = -0.24$, n = 12, CI spanning zero). An adversarial self-critique pass cleared the final claim set with zero surviving overclaims.
1 Introduction
The paper reframes long-horizon agent memory as managing decision-relevant state and studies consolidation through a progression from learning what to retain to governing abstraction. Its four-phase throughline is a set of rigorous ways to know whether consolidation decisions are sound and accountable.
1.1 From “remember more” to “remember well” to “remember accountably”
- The program moves from episode segmentation to learned abstraction, oracle-based quality measurement, and governed consolidation.Phase 1 asks where experience should be segmented; Phases 2–3 address when and how to abstract and how to score decisions; Phase 4 makes abstraction poison-resistant, reversible, and auditable.Source: §1.1 · p. 2
1.2 The throughline: epistemics
- The paper presents rigorous ways to know—not four solutions—as its durable contribution.These include checking for self-leakage, diagnosing a forgetting failure, scoring consolidation against an optimum, and auditing or constraining beliefs.Source: §1.2 · p. 2
1.3 Contributions
- The contributions include an anti-leakage protocol, a learned promotion-and-level policy, ConsolidationBench, governed consolidation, and adversarial self-critique.The governance contribution is statistically distinct from the quality metric, so it does not depend on that metric’s contested external validity.Source: §1.3 · pp. 2–3
- Some mechanism internals, gate thresholds, and benchmark-generator parameters are under controlled release, while reported numbers reproduce bit-exactly from persisted artifacts.The withheld items are implementation constants, not findings.Source: §1.3 · p. 3
2 Phase 1, Where Does a Memory Begin?
Phase 1 learns episode boundaries from agent traces using downstream task utility, but its detector is reported as a near-miss rather than a success. Its lasting contribution is a three-gate protocol intended to prevent learned-memory experiments from covertly reading answer keys.
2.1 Problem
- Phase 1 targets the gap between heuristic segmentation or learning on pre-given boundaries and learning boundaries from traces by downstream task utility.The target combines reading agent traces, learning without boundary labels, and optimizing boundaries by task utility.Source: §2.1 · p. 3
2.2 Method and result (honest near-miss)
- BoundaryLearner is trained on procedurally generated traces with planted ground-truth boundaries and optimized end-to-end by downstream reward without boundary labels.The pre-registered success criterion required oracle correlation above 0.70, more than 10% task-success gain over fixed intervals, and beating Bayesian-surprise and topic-coherence baselines.Source: §2.2 · p. 3
- 0.691 oracle correlation missed the 0.70 bar, and task success was 0.679 versus Bayesian surprise’s 0.705; the result was a reported near-miss.BoundaryLearner cleared Bayesian surprise on oracle correlation (0.691 vs. 0.315), but the pre-registered criterion was not met.Source: §2.2 · p. 3, Table 1 · p. 3
2.3 The lasting contribution: not the detector, the discipline
- The lasting output is a three-gate anti-leakage protocol: provenance checks, feature probes, and count-matched controls.The gates check for generator-internal fields, single observable features that predict latent answers above chance, and unequal cluster counts or budget pressure.Source: §2.3 · p. 4
3 Phase 2, When, and to What Level, to Abstract
Phase 2 learns when to promote experience and which abstraction level to use under a hard token budget, but its forgetting decisions remain problematic. A follow-on forgetting signal improves precision yet becomes nearly inactive under a different junk prevalence.
3.1 A verified gain, and a precisely characterized failure
- +22.7% relative task-success improvement over the best non-learned baseline on the medium configuration came with 7.17× memory compression.The result was reproduced bit-exactly and verified on ten fresh held-out seeds; the paired confidence interval excluded zero, and the compression confidence interval exceeded the pre-registered 5× bar.Source: §3.1 · p. 4
- Forgetting precision was 0.319 against a 0.333 junk base rate, and budget-pressure forgetting destroyed 74% of load-bearing clusters at least once.A budget-pressure forget → premature re-promotion → wrong artifact → transfer-poisoning chain accounted for 75% of the hard-configuration transfer regression (−15.0 pp).Source: §3.1 · p. 4
- The policy learned when to promote but not when to forget; applying the wrong abstraction can replace correct raw evidence with a falsely distilled rule.Raw experience mass exceeded the reference budget by 7.3×/9.4×; recurring parametric sub-tasks warrant skills, stable invariants rules, and one-offs neither.Source: §3.1 · p. 4
3.2 Targeting forgetting, and a second mechanistic finding
- Adding observable forgetting evidence and hindsight-regret credit raised forgetting precision to 0.932 (hard) and 0.833 (medium), but reduced forgetting to about 0.61 per seed from ∼30 and regressed overall success.The named λ–prevalence coupling failure occurred when a gate tuned at junk prevalence 0.593 collapsed at 0.388.Source: §3.2 · pp. 4–5
4 Phase 3, Measuring Consolidation Decision Quality
ConsolidationBench scores memory decisions against an exactly computable oracle on retention, compression, and transfer. Its leaderboard shows that retrieval systems can retain information while achieving no composite-task transfer.
4.1 The gap: everyone scores accuracy, nobody scores the decision
- Existing agent-memory benchmarks score downstream accuracy, confounding base-model capability, retrieval quality, and consolidation-decision quality.The Experience Compression Spectrum names a four-level hierarchy but provides no benchmark.Source: §4.1 · p. 5
4.2 ConsolidationBench: an oracle-by-construction
- ConsolidationBench procedurally generates experience streams with known cluster values, making the optimal promotion policy exactly computable while keeping oracle and latent types hidden.Policies receive only whitelisted features, and a headroom gate checks that the oracle beats the best non-learned baseline before training.Source: §4.2 · p. 5
- The benchmark scores retention, compression, and reuse against the computable optimum, combining normalized retention and reuse in the CQS with a small reuse floor.The floor grades policies with real retention rather than zeroing them when they fail to generalize.Source: §4.2 · p. 5
4.3 The leaderboard finding
- Eight of ten policies had zero composite-task transfer; literature-faithful retrieval systems retained well (oracle-normalized retention 0.53 / 0.49) yet achieved zero transfer.The leaderboard’s graded CQS separates zero-transfer policies by retention without changing the transfer finding.Source: Table 2 · p. 6, §4.3 · p. 6
- The three score axes are empirically distinct: the maximum pairwise correlation is |r| = 0.59, while compression–retention correlations are 0.17 (medium) and 0.23 (hard).These values are below the 0.9 redundancy threshold.Source: §4.3 · p. 6
5 Phase 4, Governing the Consolidation Decision
Phase 4 wraps consolidation in poison-resistance, reversibility, and auditability guarantees with a quality gate, then tests whether its quality metric predicts real transfer. The governance measures are distinct from CQS, while the external-validity study finds that CQS does not predict real transfer accuracy.
5.1 The new question: who guards the abstraction?
- The paper targets consolidation governance because a distilled false rule can silently corrupt downstream decisions, while existing memory governance focuses on storage and sharing.The stated risks are poison, irreversibility, and opacity.Source: §5.1 · p. 6
5.2 Governed consolidation
- The governed policy combines poison-resistance, rollback, and an audit ledger with a quality gate; it is the only evaluated policy satisfying both governance and quality criteria.It admitted zero false rules from planted decoys, rollback fired on 5/5 seeds with task success recovering from 0.0 to 1.0, and promotion records were complete for 100% of promotions.Source: §5.2 · p. 7, Table 3 · p. 7
5.3 The insurance property (why this survives a hostile reviewer)
- Governance and CQS are distinct: aggregate governance versus CQS0 has r² = 0.43, partial r = 0.27, and identical-CQS0 policies differ threefold in governance.The governance guarantees use measures independent of CQS: false-rule counts, task-success behavior, and ledger completeness.Source: §5.3 · p. 7
5.4 External validity: a metric redesign, and a resolved negative
- CQS redesign removed the structural ceiling: collapse fell from fifteen of twenty rows to zero, while CQS remained discriminant from retention and tracked transfer among generalizing policies.The selected geometric-mean form had ρ(CQS, retention)=0.88 and ρ(CQS, transfer)=0.90; additive aggregation was rejected because it collapsed CQS onto retention (ρ=0.98).Source: §5.4 · p. 8
- Pooled Spearman ρ = −0.24 (n=12; 95% CI [−0.79, +0.49]) does not support CQS predicting real transfer accuracy across two benchmarks.The result replicated as ρ = +0.03 on LoCoMo and ρ = −0.30 on 2Wiki; the single-hop control inverted the pattern, and all three axes failed individually.Source: §5.4 · p. 8
- CQS is an oracle-normalized measure of structural consolidation quality, not a validated predictor of downstream answer quality; matching real multi-hop QA remains future work.The authors state that its composite-task axis would need to be redefined to match how real multi-hop questions are answered.Source: §5.4 · p. 8
5.5 Adversarial self-critique
- The Phase-4 adversarial audit reviewed 17 claims and found zero surviving overclaims, while requiring three hedges about governance, rollback, and CQS validity.The audit required G4 binding, scoped the rollback result to a purpose-built contradiction variant, and labeled CQS-dependent claims as partial-validity.Source: §5.5 · p. 9
5.6 The adversarial-verification methodology, across phases
- Across phases, independent adversarial verification reapplied anti-leakage gates and caught the gameable-governance exploit that required mandatory G4 binding.The paper reports the verification outcomes in Table 4.Source: §5.6 · p. 9, Table 4 · p. 9
6 Related Work
Related work spans downstream task and skill-generation benchmarks, memory architectures, governance systems, and consolidation diagnostics. The paper positions ConsolidationBench as isolating the consolidation decision against an oracle and governed consolidation as a policy-agnostic wrapper for that decision.
Memory benchmarks
- Task-accuracy and skill-generation benchmarks measure end-to-end outcomes, whereas ConsolidationBench isolates the consolidation decision against an oracle.The paper also distinguishes existing consolidation policies from its policy-agnostic governance wrapper.Source: §6 · p. 9
Memory governance
- Prior governance work addresses storage and sharing or specifies governance desiderata, while this work measures governance at the consolidation decision.The paper distinguishes its claim from introducing memory governance or reversibility, positioning G2 as rollbackability plus verified forgetting and G3 as provenance visibility.Source: §6 · pp. 9–10
Foundations
- The paper situates boundary detection in event-segmentation cognition and prior agent-memory systems, and its four-level compression hierarchy in the Experience Compression Spectrum.It attributes the state-not-storage framing and tripartite state organization to the cognitive-state-plane formulation.Source: §6 · p. 10
7 Reproducibility and Controlled Release
The paper reports persisted, independently re-derived numerical results and describes selected implementation details under controlled release. It presents these choices as balancing verifiability with protection of an engineering artifact.
Reproducibility
- Every numerical claim traces to a persisted machine-readable result file, reproduced bit-exactly and independently re-derived.Integrity self-tests and the Phase-1 anti-leakage gates pass on a clean environment.Source: §7 · p. 10
Controlled release
- Exact gate thresholds, instability and detail-grounding signals, benchmark-generator parameters, and production integration mapping are held under controlled release.The details are available to reviewers and collaborators on request under terms.Source: §7 · p. 10
- The authors describe these withheld details as implementation constants and recipes, while stating that reported results, axes, verdicts, and the insurance property are fully specified.They frame the arrangement as balancing scientific credibility with protection of a non-trivial engineering artifact.Source: §7 · p. 10
8 Discussion: The Epistemics Throughline
The discussion frames the four-phase program as a progression in how to know whether memory consolidation is reliable. Its durable contribution is a set of methods for checking, diagnosing, measuring, and governing consolidation decisions.
8 Discussion: The Epistemics Throughline
- The program’s contributions are framed as four ways to know rather than four solutions.These address self-leakage, learned forgetting failures, consolidation quality against an optimum, and auditing or reversing learned beliefs.Source: §8 · p. 10
- Each phase’s results became a foundation for the next: anti-leakage gates protected policy claims, failure diagnosis shaped the benchmark, and the benchmark made governance measurable.Source: §8 · p. 10
- The paper identifies rigorous ways to know whether memory is solved as its durable bet, beyond any single result.Source: §8 · p. 10
9 Limitations
The reported limits distinguish CQS as a consolidation-decision score, not a predictor of downstream answer quality, and constrain the scope of the governance and benchmark evidence.
9 Limitations
- Pooled Spearman ρ = −0.24 (n=12; CI spanning zero) does not support CQS as a predictor of downstream answer quality.The paper states that CQS should not be cited as a predictor.Source: §9 · p. 10
- G2 rollback is demonstrated on a constructed variant stronger than the main benchmark’s break pattern; a harder natural-contradiction test remains.Source: §9 · p. 11
- ConsolidationBench is synthetic-first: holding transforms and base-model capability constant trades ecological validity for diagnostic power.Source: §9 · p. 11
- Governance has been verified on one consolidation policy, although the wrapper is policy-agnostic in principle.Source: §9 · p. 11
- Phase 1’s detector did not clear its bar, and the paper reports it as a negative result.Source: §9 · p. 11
10 Conclusion
The conclusion presents the program’s outcome as tools for assessing and governing what agents decide to believe. It emphasizes making abstraction auditable, reversible, and poison-resistant.
10 Conclusion
- The program ends with tools for judging whether an agent’s memory can be trusted, rather than memory alone.The tools include an anti-leakage discipline, a failure-diagnosis method, an oracle-grounded quality benchmark, and a governed abstraction mechanism.Source: §10 · p. 11
- Governed abstraction is auditable, reversible, and poison-resistant, and is reported to stand independently of the quality metric’s contested external validity.Source: §10 · p. 11
- The conclusion frames accountability as asking what an agent has decided to believe and whether that decision can be held accountable.Source: §10 · p. 11