Source-linked AI summary
Dual-Layer Agentic Memory with Fast Write Routing and Slow Consolidation
Wenzhi Li, Dong Nie, Rui Lan, Tongtong Lyu, Peiyao Wang, Lingzi Hong, Weihang Pan, Boyuan Pan, Yao Hu
TL;DR
LLM agents need memory that manages evolving knowledge without bloating retrieval or relying on costly, forgetting-prone parameter updates. The paper proposes Dual-Layer Agentic Memory, combining cost-aware write routing with periodic supervised consolidation. Its cascade reduces routing and storage burdens while retaining near-exhaustive-memory QA performance, and consolidation enables adaptive suppression of redundant writes.
Problem
Existing memory systems often append external knowledge monotonically, while parametric updating is costly and susceptible to catastrophic forgetting.
Method
Dual-Layer Agentic Memory uses a write-phase taxonomy and cost-aware small-to-large admission cascade, followed by periodic supervised fine-tuning of selected external memories.
Results
The cascade escalates 39.7% to 49.0% of uncertain queries, while consolidation achieves 90.71% QA EM at 32.08% storage footprint, retaining over 98.2% of the Full Store upper bound.
Takeaways & Limitations
The results support selective externalization followed by selective internalization as a unified memory lifecycle for LLM agents.
Takeaways & Limitations
Offline SFT write-back introduces internalization latency and high compute costs, while conflicting updates and catastrophic forgetting during consolidation remain unresolved.
Abstract
from arXiv · showhide
Large language model (LLM) agents operate in dynamic environments where knowledge continuously evolves. Existing memory systems typically treat external memory as a monotonically growing repository, inevitably leading to retrieval degradation and increasing computational costs over time. We argue that the core challenge is not retrieval alone, but managing the knowledge lifecycle: deciding what to externalize, update, or ultimately internalize. Inspired by Complementary Learning Systems (CLS) theory in neuroscience, we propose Dual-Layer Agentic Memory, a framework that shifts memory management to the write phase through cost-aware epistemic routing and periodic parametric consolidation. Incoming information is categorized as non-write, write-new, or write-update, and routed through a small-to-large model cascade that minimizes routing overhead while filtering redundant memories. A subsequent write-back phase selectively consolidates high-value external memories into model parameters via supervised fine-tuning. Experiments demonstrate the dual efficiency of our approach: a 1.7B/8B cascade prunes up to 68% of redundant external memory while escalating fewer than 50% of inputs, yet retains over 98% of the downstream QA Exact Match (EM) achieved by an exhaustive retention baseline. We further show that periodic consolidation successfully internalizes external knowledge, allowing the router to adaptively suppress redundant writes as the model's epistemic boundaries evolve. Overall, our framework presents a unified paradigm for agent memory: selective externalization followed by selective internalization. Code and dataset will be released upon acceptance.
1 Introduction
The paper reframes agent memory as a knowledge lifecycle: selectively externalize information during writing, then periodically internalize valuable memories into model parameters. A cost-aware cascade routes admissions while consolidation reduces future reliance on external memory.
- Motivation: Existing external-memory systems append observations monotonically, while parametric updates are costly and prone to catastrophic forgetting.These approaches treat memory as a fixed storage substrate rather than dynamically allocating, revising, and migrating knowledge.
- Motivation: The framework treats external memory as a selective, temporary store for knowledge missing from parameters or conflicting with stale beliefs.This shifts filtering from query-driven retrieval to decisions made when new facts enter the system.
- Framework: The operational taxonomy distinguishes non-write, write-new, and write-update cases, while the router makes a cost-aware binary write-or-discard admission decision.Write-new denotes missing knowledge, write-update indicates a confident but incorrect answer, and non-write denotes knowledge already answerable from parameters.
- Routing: The small-to-large cascade uses a lightweight model for cheap screening and escalates only uncertain items to a stronger model.This design treats routing as resource allocation so the admission process does not become too expensive.
- Consolidation: Periodic supervised fine-tuning internalizes selected memories, allowing the router to suppress future writes for knowledge absorbed into parametric memory.The write-back phase targets selected write-new and write-update memories and adapts to shifted internal representations.
- Framework: Dual-Layer Agentic Memory combines parametric and external memory with fast write routing and slow write-back consolidation.Parametric memory stores stable compressed knowledge, whereas external memory holds dynamic, editable, high-risk facts.
2 Related work
Related work extends agent memory through external retrieval and routing, parametric updating, and neuroscience-inspired consolidation. The paper distinguishes its approach by shifting memory-admission decisions to the write phase while using external memory as an editable buffer.
- Memory architectures in LLM-based agents: Long-horizon agents commonly use external RAG and memory-management systems, with newer work improving compression, linking, and structured representations.These approaches extend working memory beyond the context window.
- Retrieval-augmented generation and routing: Recent routing frameworks mainly operate during reading, whereas this paper targets the earlier decision of what to externalize before storage.The distinction is between query-conditioned retrieval routing and write-phase memory admission.
- Knowledge editing and continual parametric updating: Knowledge-editing and continual-learning methods update parameters but remain computationally expensive and susceptible to catastrophic forgetting.The paper motivates reserving parameter modifications for consolidated knowledge while using external memory for fast edits.
- Neuroscience-inspired memory and consolidation: CLS-inspired systems model interaction between fast-learning hippocampal and slow-learning neocortical processes through consolidation mechanisms.This neuroscience framing motivates the paper’s two-layer memory design.
3 Dual-Layer Agentic Memory
The framework treats agent memory as a dual-layer lifecycle: a fast router selectively admits knowledge to external memory, while slow write-back consolidates validated memories into model parameters. Routing uses behavioral labels, cost-aware small-to-large escalation, and frozen-backbone features to balance task performance, storage, and computation.
- Dual-layer memory: The system maintains stable, compressed parametric memory alongside explicit, editable external memory for dynamic or conflict-prone facts.Its decision layer separates fast write routing from slow write-back consolidation.
- Operational memory taxonomy: The operational taxonomy distinguishes non-write, write-new, and write-update according to zero-shot and memory-supported answering behavior.Non-write means parametric memory already answers correctly; write-new indicates refusal from missing knowledge, while write-update indicates a confident but incorrect stale belief.
- Lifecycle decisions: At storage time, the router makes a binary discard-or-write decision, and written items later update the model to Θ′ through slow write-back.External items are used for parametric consolidation rather than retained indefinitely by default.
- Cost-aware routing: The small-to-large cascade screens inputs cheaply and escalates only uncertain or ambiguous cases to the stronger router.The gate escalates when predicted escalation gain is positive, with escalation penalized by λe.
- Cost-aware routing: The routing objective rewards writing when downstream QA Exact Match gain exceeds the storage penalty, while escalation incurs an additional compute penalty.The write reward is r(W) = EM(W) − EM(D) − λs, and the discard reward is r(D) = 0.
- Feature construction and parameterization: Frozen LLM backbones provide semantic and uncertainty features to MLP decision heads, which are trained to regress reward targets while tracing EM–storage–compute trade-offs.Features include mean-pooled hidden states and token-level negative log-likelihood representations.
- Write-back consolidation: Write-back converts routed facts into factual-probe supervision and applies only to write-new or write-update memories.A fact is removed from external memory only after the updated model answers its probes correctly without retrieval; otherwise it remains external.
4 Experiments and results
The experiments evaluate online memory admission and consolidation, showing that cascaded write routing preserves near-exhaustive QA performance while reducing routing cost and external storage. Write-back further internalizes knowledge and changes which facts require external memory.
- Benchmark and setup: The benchmark evaluates continuous memory management with streaming episodes, temporally interleaved fact injections, downstream QA, storage decisions, and routing compute.The evaluation uses E = 300 episodes with T = 500 turns and K = 100 candidate fact injections per episode.
- Benchmark and setup: The benchmark compares static, heuristic, supervised, cascaded, and write-back policies against exhaustive storage as a theoretical upper bound.Methods include No Store, Full Store, Random Store, heuristic policies, Logistic Regression, MLP Classifier, Write Router, and Write RouterSFT.
- Cascade routing: 39.7% to 49.0% of uncertain queries escalate to the 8B model, reducing expected inference cost by roughly 30% to 39%.The small-to-large cascade uses qwen3-1.7B as a proxy before selectively invoking qwen3-8B.
- Cascade routing: Over 98.3% of the Full Store EM upper bound is retained while the Write Router reduces external memory bloat and routing compute overhead.The table caption summarizes the cascade as escalating approximately 40–49% of inputs, pruning up to 68% of redundant memory, and retaining over 98.2% of Full Store EM.
- Write-back and consolidation: Write RouterSFT achieves 90.71% QA EM with 97.85% Store Precision while reducing the storage ratio to 32.08%.Periodic SFT internalizes knowledge, allowing the router to discard newly internalized facts and target remaining blind spots.
- Write-back and consolidation: 67.46% of held-out knowledge-base labels change after SFT, including 10,683 retrieval-dependent facts becoming non-write and 1,752 stable facts degrading to write-update.The transitions show both parametric internalization and parametric interference after consolidation.
5 Discussion & conclusion
The paper frames agent memory as a dynamic knowledge lifecycle combining selective externalization with parametric consolidation. It concludes that external memory remains necessary because consolidation can internalize knowledge while also causing interference, and it identifies latency, conflict resolution, and forgetting as future challenges.
- Discussion and conclusion: The framework treats LLM agent memory as a dynamic knowledge lifecycle rather than a passive, monotonically growing repository.Its architecture combines fast selective write routing with slow parametric consolidation, inspired by complementary learning systems.
- Discussion and conclusion: Selective externalization preserves downstream accuracy near the theoretical upper bound while reducing external memory bloat and routing compute overhead.The conclusion attributes this result to a small-to-large cascaded router that filters redundant facts.
- Discussion and conclusion: Write-back uses external memory as a temporary buffer for internalizing retrieval-dependent facts through periodic SFT.The router adapts to the resulting shift in parametric capacity and further suppresses the external memory ratio.
- Discussion and conclusion: The external layer remains necessary because continuous parametric updates can disrupt a subset of previously stable knowledge.The paper presents fast-routing external memory as a buffer that can correct epistemic regressions caused by consolidation.
- Limitations and future work: Offline SFT write-back introduces internalization latency and high compute costs, while uniform-buffer consolidation lacks principled handling of temporally conflicting updates.The paper also identifies active mitigation of catastrophic forgetting during write-back as an unresolved challenge.
A.2 Streaming evaluation protocol and episode synthesis
The benchmark synthesizes finite-horizon streaming episodes in which injected facts and downstream queries co-occur under coverage, causality, liveness, and distractor-exposure constraints. A deterministic three-stage pipeline schedules injections, allocates probes, and streams only causally valid queries while evaluating QA and storage decisions.
- 300 episodes each contain 500 turns and 100 private-pool facts drawn without replacement from the test split.
- The scheduler enforces coverage, causality, liveness, and distractor exposure, including natural-rate non-write distractors.
- A seeded three-stage pipeline stratifies fact injections, allocates probe budgets to fill the episode length, and streams probes only after their facts are injected.
- O(T log T) synthesis provides bit-exact reproducibility while respecting the benchmark’s temporal and causal requirements.
- Evaluation reports QA Exact Match, token-level F1, refusal rate, storage ratio, and Store Precision, Recall, and F1.
B Implementation of baselines and compared methods
The implementation compares the proposed memory decisions with heuristic, surprisal-based, and learned storage baselines, alongside a deterministic episode-synthesis algorithm. The baselines differ in whether they probe explicitly, threshold perplexity, or learn binary write decisions.
- Algorithm 1 deterministically synthesizes episodes through stratified injection, probe-budget allocation, and causal probe streaming.
- The heuristic baseline probes each incoming fact, answers a generated factual question, and stores the fact using a confidence judgment.
- PPL-Conditional Store thresholds per-token cross-entropy on the fact, using linguistic surprisal as a proxy for prior familiarity.
- Logistic regression and MLP classifiers learn binary write versus non-write labels, with the MLP matching the large router architecture and features but using BCE.
C Training and deployment protocols
The routers are trained stagewise on reward targets, after which a gate learns when escalation is valuable. Deployment selects routing policies on a validation Pareto frontier rather than optimizing QA EM alone.
- Small and large routers regress reward targets with mean squared error before the gate is trained from their greedy actions.
- The gate predicts escalation gain after both routers are frozen.
- Validation sweeps over λs and λe trace the Pareto frontier because maximizing QA EM alone would favor writing everything.
D.1 Feature ablation
Feature ablations show that semantic inputs are stronger than uncertainty-only inputs for the small router, while adding small-router features gives the large router little benefit. The resulting comparisons quantify storage required at matched test EM or storage caps.
- At test EM 0.86, single_small_semantic requires storage 0.710 versus 0.847 for single_small_uncertainty.
- At storage ≤0.8, single_large_semantic reaches test EM 0.874 versus 0.875 for single_small_large_semantic.
- At EM ≥0.86, the required storage is 0.658 for single_large_semantic versus 0.662 for single_small_large_semantic.
D.2 Pooling and loss ablation
The ablation compares pooling strategies and training losses for the routing model on a held-out test set. Mean pooling with MSE provides the strongest overall storage–accuracy trade-off.
- Mean pooling with MSE yields the strongest overall Pareto frontier in the pooling-and-loss ablation.The comparison evaluates mean versus last-token pooling and MSE versus margin-ranking objectives.
- 0.875 test EM is achieved by mean+mse at storage ≤ 0.8, narrowly exceeding mean+margin_ranking at 0.874.
- 0.658 storage is required for mean+mse to reach EM ≥0.86, compared with 0.680, 0.678, and 0.690 for the other configurations.The alternatives are mean+margin_ranking, last_token+mse, and last_token+margin_ranking, respectively.
D.3 Escalation thresholds analysis
The escalation-threshold analysis measures how closely thresholded small-to-large cascades approximate the large-router frontier while improving on the small router under storage caps.
- 0.0004, 0.0022, and 0.0010 EM are the ρ ≤0.6 cascade deficits relative to the large router at storage caps 0.6, 0.7, and 0.8.
- 0.0120, 0.0048, and 0.0021 EM are the corresponding improvements over the small-router frontier.
- 0.0031 is the maximum gap to the large-router frontier under the stricter ρ ≤0.4 budget across the same storage range.
D.4 Cascade gate design ablation
The cascade-gate ablation evaluates learned gate designs under two escalation thresholds. The semantic-uncertainty gate most closely matches the large-router frontier and reaches target EM with the lowest storage under the stricter budget.
- Cascade gate design ablation: 0.8743 test EM is reached by cascade_gate_semantic_uncertainty at storage ≤0.8 under ρ ≤0.6, only 0.0003 below the large-router frontier.
- Cascade gate design ablation: 0.6267 storage is required for cascade_gate_semantic_uncertainty to reach EM ≥0.85 under ρ ≤0.4, versus 0.6464 and 0.6766 for the other gates.The alternatives are cascade_gate_semantic and cascade_gate_score_only, respectively.
- Cascade gate design ablation: 0.8451 and 0.8656 test EM are achieved by cascade_gate_semantic at storage ≤0.6 and ≤0.7 under ρ ≤0.6.
- Cascade gate design ablation: 0.8428 and 0.8638 are the corresponding scores for cascade_gate_score_only at the same storage caps.