Source-linked AI summary
$δ$-mem: Efficient Online Memory for Large Language Models
Jingdi Lei, Di Zhang, Junxian Li, Weida Wang, Kaixuan Fan, Xiang Liu, Qihan Liu, Xiaoteng Ma, Baian Chen, Soujanya Poria
TL;DR
Long-term assistants and agents need to accumulate and reuse historical information, but context expansion is costly and may not ensure effective use. δ-mem adds a compact online associative-memory state to a frozen attention backbone, improving average scores to 1.10× those of the backbone and 1.15× those of the strongest non-δ-mem baseline.
Problem
Long-term assistants and agents need to accumulate, update, and effectively reuse historical information during extended interactions.
Method
δ-mem compresses past information into a dynamically updated fixed-size associative-memory state whose readout produces low-rank corrections to frozen-backbone attention.
Results
1.10× average-score improvement over the frozen backbone and 1.15× over the strongest non-δ-mem memory baseline were achieved with an 8×8 online state, with larger gains on memory-heavy tasks.
Takeaways & Limitations
A compact online state directly coupled to attention can provide effective associative memory while largely preserving general capabilities.
Takeaways & Limitations
δ-mem trains its parameters while keeping the backbone parameters frozen.
Abstract
from arXiv · showhide
Large language models increasingly need to accumulate and reuse historical information in long-term assistants and agent systems. Simply expanding the context window is costly and often fails to ensure effective context utilization. We propose $δ$-mem, a lightweight memory mechanism that augments a frozen full-attention backbone with a compact online state of associative memory. $δ$-mem compresses past information into a fixed-size state matrix updated by delta-rule learning, and uses its readout to generate low-rank corrections to the backbone's attention computation during generation. With only an $8\times8$ online memory state, $δ$-mem improves the average score to $1.10\times$ that of the frozen backbone and $1.15\times$ that of the strongest non-$δ$-mem memory baseline. It achieves larger gains on memory-heavy benchmarks, reaching $1.31\times$ on MemoryAgentBench and $1.20\times$ on LoCoMo, while largely preserving general capabilities. These results show that effective memory can be realized through a compact online state directly coupled with attention computation, without full fine-tuning, backbone replacement, or explicit context extension.
1 Introduction
The introduction frames long-term assistants and agents as requiring models to accumulate, update, and reuse historical information effectively during extended interactions. It presents δ-mem as a compact, dynamically updated associative memory coupled to a frozen full-attention backbone, achieving gains with only an 8 × 8 state.
- Motivation: Long-term assistants and agents must accumulate, update, and reuse historical information across extended, memory-heavy interactions.The memory must also remain dynamically maintained and usable by the backbone during test-time.
- Related work: Prior memory mechanisms trade off modularity, integration overhead, and dynamic adaptation under a fixed context window.Outside-channel mechanisms add retrieval or encoding overhead and possible misalignment, while parametric mechanisms are static despite compatibility with frozen backbones.
- Method: δ-mem augments a frozen full-attention backbone with a compact online associative memory state updated by delta-rule learning.It compresses past information into a fixed-size matrix rather than storing all historical tokens in the input context.
- Results: 8 × 8 online state of associative memory yields a final average score 1.10× over the frozen backbone and 1.15× over the strongest non-δ-mem memory baseline.The evaluation covers HotpotQA, LoCoMo, MemoryAgentBench, IFEval, and GPQA-Diamond.
- Results: An 8 × 8 matrix retains useful historical signals and helps recover context-relevant information after explicit history is removed.The reported gains occur without full fine-tuning or replacing the backbone architecture.
2 Preliminaries
δ-mem maintains a compact associative-memory state that reads history to steer frozen-backbone attention, then updates online through gated delta-rule learning. Its design projects hidden states into a low-dimensional memory space, supports dimension-wise retention and writing, and can write at token or message granularity.
- Gated delta-rule update: The online state uses residual error correction with controlled forgetting: βt scales residual writing, while λt controls retention of previous memory.Dimension-wise gates allow some memory dimensions to retain old information while others write current information more actively.
- Associative memory projections: δ-mem projects each hidden state into a low-dimensional associative-memory space, producing query and key vectors for reading and writing the online state.Normalizing the query and key reduces instability from scale drift during long-sequence recurrence.
- Attention steering: Read signals generate low-rank query-side and output-side corrections to the frozen backbone’s attention computation, with effects varying by history through the dynamic state.The main implementation applies corrections on the query and output sides; the correction parameters are fixed after training.
- Memory reading: The read vector queries the fixed-size previous state, providing continuous history-dependent signals without returning text segments or adding context tokens.Because the state size is fixed, read cost is independent of history length.
- Writing granularity: δ-mem examines token-level TSW and message-level SSW writing strategies, trading fine-grained local information against smoother state evolution and fewer redundant writes.TSW is more sensitive to formatting symbols, repetition, and short-term noise, whereas SSW can absorb fine-grained details through averaging.
4 Experiments
Experiments evaluate δ-mem against textual and parametric memory mechanisms on general reasoning, instruction-following, and memory-heavy benchmarks. δ-mem achieves the strongest overall results, with especially substantial gains on memory-heavy tasks and consistent improvements across backbone models.
- Evaluation and baselines: δ-mem achieves the strongest performance across representative memory-augmented methods on general reasoning, instruction following, and memory-heavy benchmarks.All methods use the Qwen3-4B-Instruct backbone.
- Main results: 51.66% is the best average score, achieved by TSW, improving over the Qwen3-4B-Instruct backbone at 46.79% and Context2LoRA at 44.90%.These correspond to improvements of +4.87 and +6.76 points, respectively.
- Memory-heavy benchmarks: 38.85% is δ-mem’s average score on MemoryAgentBench, up from 29.54%, while MSW reaches 49.12% on LoCoMo and leads its Multi, Temporal, and Open subsets.On HotpotQA, TSW improves EM/F1 from 42.35%/56.00% to 49.41%/63.66%.
- Baseline analysis: Textual memory methods show inconsistent gains, while Context2LoRA generalizes less robustly and MLP Memory performs relatively limited.The reported limitations are attributed to retrieval noise, token-space information loss, static parameter encoding, and insufficient sequential memory.
- Backbone evaluation: 51.66% is δ-mem’s Qwen3-4B-Instruct average, compared with 50.86% on Qwen3-8B and 36.96% on SmolLM3-3B.The corresponding backbone scores are 46.79%, 47.20%, and 26.08%, respectively.
5 Ablative Study
The ablation study identifies effective interfaces and depths for injecting δ-mem corrections. Query/output branch injection performs strongly, while applying corrections across all layers yields the best reported overall and task-specific results.
- Head Ablation: Applying δ-mem to both query and output branches yields strong performance, indicating effective interfaces for memory injection.The study identifies query-side and output-side corrections as effective attention-block interfaces.
- Head Ablation: 47.05% average score is achieved by the output-only branch, outperforming the less-effective key-only variant.Among single-branch variants, the output branch performs best.
- Insertion Depth: 47.97% average score is achieved when memory correction is applied to all layers, the best overall performance.All-layer insertion also produces the strongest reported HotpotQA and LoCoMo results.
- Insertion Depth: 49.41%/63.66% overall EM/F1 is obtained on HotpotQA with all-layer correction.This is the strongest HotpotQA result reported in the insertion-depth study.
- Insertion Depth: 46.53% LoCoMo average is reached with all-layer correction, the best result in the insertion-depth study.The results suggest memory signals can influence the representation hierarchy across the backbone’s full depth.
6 Related Work
Related work organizes memory mechanisms into textual, outside-channel latent, and parametric approaches. δ-mem instead maintains a compact online state that steers a frozen Transformer through low-rank attention corrections without reinserting compressed history into token space.
- Textual Memory Mechanisms: Textual memory externalizes entries, summaries, or documents and reinjects selected evidence through input context or retrieval-augmented generation.Continuous-interaction systems extend this paradigm through logging, summarization, and reflection.
- Textual Memory Mechanisms: δ-mem avoids token-space reinsertion by maintaining a compact online state whose readout supplies low-rank attention corrections to a frozen Transformer.This separates memory maintenance from prompt-level reinsertion and avoids sensitivity to compression fidelity, retrieval noise, and context budget.
- Outside-Channel Memory Mechanisms: Outside-channel mechanisms preserve memory in latent form outside the backbone, including non-differentiable key-value memories retrieved by approximate kNN and adaptive residual side networks.Memorizing Transformers and LongMem exemplify these approaches.
- Parametric Memory Mechanisms: Parametric mechanisms encode memory in additional parameters or localized weight edits, including continuous virtual tokens, low-rank updates, and model-editing methods.Prefix-Tuning and LoRA demonstrate small parameter additions for steering frozen models, while ROME and MEMIT treat parameters as writable memory.
7 Conclusion
δ-mem is a lightweight memory mechanism that equips a frozen full-attention backbone with a compact, dynamically updated online associative-memory state. It compresses past information into fixed-size state and uses readout-generated low-rank attention corrections, enabling online memory without full fine-tuning or backbone replacement.
- Conclusion: δ-mem equips a frozen full-attention backbone with a compact, dynamically updated online state of associative memory.The mechanism is designed as a lightweight addition to the existing backbone.
- Conclusion: δ-mem compresses past information into a fixed-size online state and uses its readout to generate low-rank corrections to attention components.The state is directly coupled to the backbone’s attention computation.
- Conclusion: Online memory participates directly in forward computation without full fine-tuning or replacing the backbone architecture.This design supports memory maintenance during online operation while preserving the frozen backbone.
Appendix · A Implementation Details
The appendix specifies the training configuration, online-memory settings, hardware, precision, and benchmark evaluation procedures for δ-mem. Evaluation follows official prompts and decoding settings, with task-specific metrics and a sample-weighted MemoryAgentBench score.
- A Implementation Details: Training uses one epoch on QASPER’s shortest 2,219-sample split, with sequences up to 8,269 tokens.The backbone training sequence length is 512 tokens.
- A Implementation Details: The memory write budget is 8,192 tokens, while the maximum backbone training sequence length is 512.
- A Implementation Details: δ-mem uses r = 8 and α = 16 by default and applies only to the query and output branches.
- A Implementation Details: The MSW configuration uses 4 states and training runs on 8 × A800 GPUs with bfloat16 precision.
- A Implementation Details: Evaluation follows official benchmark prompts and decoding settings, reporting strict accuracy for IFEval, EM and F1 for HotpotQA, accuracy for GPQA, and F1 for LoCoMo.
- A Implementation Details: MemoryAgentBench evaluates each dataset with its corresponding metric, and its final score is a sample-weighted average.
B Inference Efficiency and Memory Use
δ-mem maintains nearly the same GPU memory usage as Vanilla and Context2LoRA, even as prompt length reaches 32K. Compared with MLP Memory and MemGen, it offers a lightweight computational footprint and practical inference-efficiency balance.
- GPU memory usage: δ-mem uses nearly the same GPU memory as Vanilla and Context2LoRA at prompt lengths up to 32K.Its compact recurrent state introduces negligible memory overhead as prompts grow.
- GPU memory usage: MLP Memory and MemGen require substantially more memory than δ-mem, reflecting larger auxiliary memory representations.Their higher memory use arises from maintaining or generating larger auxiliary representations.
- Overall inference efficiency: δ-mem improves long-context memory with a lightweight computational footprint and a practical balance between memory capability and inference efficiency.This conclusion follows the comparisons across varying prompt and decoding lengths.
C Parameter Overhead
δ-mem adds substantially fewer trainable parameters than representative memory-augmented baselines. Its SSW and TSW variants use 0.12% of the backbone parameters, while MSW uses 0.48%.
- 4.87M trainable parameters are required by δ-mem’s SSW and TSW variants, equal to 0.12% of the backbone parameters.Both variants introduce the same parameter overhead.
- 19.47M trainable parameters are required by δ-mem’s MSW variant, corresponding to 0.48% of the backbone.MSW maintains multiple memory states.
- 46.20M trainable parameters are used by MemGen, whereas MLP Memory requires 3078.00M parameters, reaching 76.40% of the backbone scale.The comparison covers trainable parameter overhead across memory-augmented methods.