Source-linked AI summary
D$^2$ACCI: A Dual-Loop Diagnostic Protocol for Evidence-Preserving Agent Memory
Xule Liu, Yijun Liu, Chao Li, Shao Kun
TL;DR
Evaluating persistent-memory agents is difficult because aggregate results do not localize errors or show whether updates regress protected task slices. D2ACCI addresses this with paired statistics, slice monitoring, and stage traces, distinguishing validated improvements from null changes while achieving 98–100% DCR@3.
Problem
Persistent-memory pipelines lack reliable stage-level diagnostics and regression-aware evidence for evaluating evolving agent-memory changes.
Method
D2ACCI uses dual runtime and diagnostic loops to promote, flag, or reject memory interventions based on paired statistics, protected-slice checks, and trace localizability.
Results
1.92–3.67 percentage-point gains were statistically significant for three accepted components, while diagnostic artifacts achieved 98–100% DCR@3 across the evaluated benchmarks.
Takeaways & Limitations
The results support using paired, slice-level diagnostics and stage traces rather than aggregate wins alone for iterating memory systems.
Takeaways & Limitations
Cross-benchmark replication of some accepted components was not yet complete, and published external numbers were treated as reference context rather than controlled baselines.
Abstract
from arXiv · showhide
Memory is a key capability of LLM agents. Persistent memory extends this across sessions---enabling recall, revision, and personalization. Yet its multi-stage pipeline (ingestion, retrieval, filtering, generation) makes failures difficult to localize: end-to-end evaluation reveals that an error occurred, but not which stage caused it. Existing evaluations often report aggregate performance without paired statistical comparisons, slice-level non-regression checks, or stage-level diagnostic traces. We propose D$^2$ACCI (Diagnostic-Driven Artifact-based Closed-loop Controlled Iteration), a dual-loop protocol whose outer diagnostic gate promotes, feature-flags, or rejects memory interventions based on paired evidence, protected-slice monitoring, and trace-level localizability. We further introduce DCR, a graded observability metric that measures whether failures remain localizable, and D$^2$ACCI-Eval, a reusable artifact for gate replay. We instantiate the protocol in MemStack and evaluate on three public benchmarks, achieving 93.59% on LoCoMo, 90.93% on LongMemEval, and 57.20% on PersonaMem-V2. Five paired ablations show that supplement extraction, session-memory retrieval, and Forget Guard yield statistically significant gains (+1.9 to +3.7pp, all p $\le$ .003). In contrast, BM25/RRF is retained as a monitored feature flag---a distinction invisible to aggregate-only evaluation. A diagnostic audit shows enriched traces substantially improve root-cause agreement over result-only relabeling. Diagnostic artifacts reach 98--100% DCR@3 versus 0% for results-only logs. These results establish that robust memory-system iteration demands traceable, statistically grounded, and regression-aware evidence---exactly the gap D$^2$ACCI fills.
Introduction
Persistent-memory agents require evaluation that can localize failures and prevent slice-level regressions, not merely report aggregate accuracy. D2ACCI addresses this gap with a dual-loop, evidence-preserving protocol, diagnostic artifacts, and a MemStack evaluation demonstrating benchmark and ablation results.
- Motivation: Persistent-memory failures can arise at extraction, consolidation, retrieval, filtering, or prompt-assembly stages, while end-to-end evaluation identifies errors without localizing their causes.These failure modes make it difficult to determine which improvement should be applied.
- D2ACCI protocol: D2ACCI uses an inner runtime loop and an outer diagnostic-evolution loop to promote, feature-flag, or reject memory changes using paired evidence, protected-slice checks, and trace sufficiency.Its decision contract binds these criteria into acceptance decisions that aggregate-only evaluation cannot provide.
- Diagnostic artifacts: DCR measures whether failures remain localizable at the relevant stage, while D2ACCI-Eval preserves positive and null findings and supports paired analysis, slice gates, discordance export, and deterministic gate replay.The artifact enables later iterations to reuse prior evidence.
- MemStack case study: 93.59% on LoCoMo, 90.93% on LongMemEval, and 57.20% on PersonaMem-V2 are achieved by the MemStack instantiation across three public benchmarks.MemStack is a diagnosable memory kernel with multi-granularity storage and feature-gated retrieval.
- MemStack case study: +2.71, +3.67, and +1.92 percentage points are the gains from supplement extraction, session-memory retrieval, and Forget Guard, respectively, across five paired feature-gated configurations.The cited passage reports these as statistically significant gains; BM25/RRF yields no statistically significant gain in the supplied text.
D2ACCI Framework
D2ACCI separates inference-time memory processing from an outer diagnostic loop that evaluates candidate changes using paired evidence, protected slices, and stage-level traces. Its gate can promote, monitor, reject, or feature-flag interventions while preserving deterministic replay and failure localization.
- Diagnostic protocol: Paired comparisons partition samples into improved, regressed, both-wrong, and both-correct sets, then label the earliest actionable failure stage for errors.Failure labels include ingestion miss, retrieval miss, context-assembly error, constraint error, generation error, component hurt, and unresolved ambiguity.
- Dual-loop architecture: D2ACCI separates an inner memory loop from an outer diagnostic loop that compares feature-bearing runs against ablations and inspects statistical, slice, and trace evidence.The inner loop covers memory extraction, storage, retrieval, constraints, context assembly, and generation; the outer loop evaluates paired artifacts.
- Gate decisions: The gate rejects significant harm, feature-flags insufficient traces or inconclusive evidence, accepts with monitoring for protected-slice regressions, and accepts positive paired evidence.Protected slices are pre-specified from benchmark category structure before gate evaluation, and the decision contract archives the evidence, rationale, checks, and monitors.
- Replay and automation: Gate replay is deterministic under fixed thresholds and pre-specified protected slices, enabling repeatable evaluation of candidate memory-policy changes.The gate can be triggered automatically after each candidate run once the system is instrumented.
- Implementation principles: Three design choices enable stage-level localization: layer separation, per-run feature flags, and trace emission at every stage boundary.These choices support paired ablations and distinguish gains from better evidence versus lucky prompt formatting.
D2ACCI-Eval and Diagnostic Coverage
D2ACCI-Eval makes memory interventions auditable by combining paired aggregate and slice comparisons with schema-validated stage traces and replayable gate decisions. Diagnostic coverage improves root-cause consistency, while failure-boundary analysis localizes errors across ingestion, retrieval, constraints, and generation.
- D2ACCI-Eval: D2ACCI-Eval joins paired examples by stable IDs, computes aggregate and slice-level deltas, exports discordance sets, and supports replayable evidence-based gate decisions.Its actionability predicate requires stage-specific evidence rather than free-form labels.
- Gate diagnostics: The dual-loop gate treats BM25/RRF as null and monitored rather than promotable, while accepted components receive explicit slice monitors against silent regressions.Metric-only evaluation gives contradictory BM25/RRF verdicts across benchmarks; the dual-loop policy avoids unconditional promotion.
- Diagnostic coverage: κ=0.571 with enriched traces versus κ=0.258 from result-only information, showing that trace availability materially improves diagnostic consistency.The 60-case audit compares enriched retrieval, assembly, and constraint traces against answer-and-score-only information.
- Diagnostic coverage: 99.47% DCR@3 accompanies Supplement’s +2.71pp gain over 1540 pairs, while traces identify preserved temporal evidence that result-only outputs cannot explain.The Gina dance-class example localizes the date correction to supplement extraction.
- Failure boundary analysis: 57.1% of 2,140 wrong PersonaMem-V2 cases localize to generation/option-scoring, followed by 29.8% constraint handling, 13.0% retrieval misses, and 0.1% ingestion misses.Trace analysis identifies where memory-only improvements encounter diminishing returns.