PaperlayerView exact revision on arXiv

Source-linked AI summary

$δ$-mem: Efficient Online Memory for Large Language Models

Jingdi Lei, Di Zhang, Junxian Li, Weida Wang, Kaixuan Fan, Xiang Liu, Qihan Liu, Xiaoteng Ma, Baian Chen, Soujanya Poria

arXiv:2605.12357v1cs.AI

TL;DR

Long-term assistants and agents need to accumulate and reuse history, motivating memory that updates dynamically and works closely with backbone attention. δ-mem adds a compact online associative-memory state to a frozen full-attention model; with an 8×8 state, it reaches 1.10× the frozen backbone’s average score and 1.15× the strongest non-δ-mem memory baseline.

  • Problem

    Existing memory approaches motivate a need for a compact, dynamically evolving state that steers the backbone through a pathway aligned with its attention computation.

    Source: §1 · p. 2

  • Method

    δ-mem maintains a fixed-size associative-memory state updated online and reads it to produce low-rank corrections to a frozen backbone’s attention during generation.

    Source: §1 · p. 2

  • Results

    1.10× the frozen backbone’s final average score and 1.15× the strongest non-δ-mem memory baseline; gains are larger on memory-heavy benchmarks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    An 8×8 online state can recover useful historical information after explicit context is removed, without full fine-tuning or replacing the backbone.

    Source: §7 · p. 10

    The training setup freezes backbone parameters ϕ while leaving δ-mem parameters θ trainable.

    Source: §3.6 · p. 6

Abstract

from arXiv · show

Large language models increasingly need to accumulate and reuse historical information in long-term assistants and agent systems. Simply expanding the context window is costly and often fails to ensure effective context utilization. We propose $δ$-mem, a lightweight memory mechanism that augments a frozen full-attention backbone with a compact online state of associative memory. $δ$-mem compresses past information into a fixed-size state matrix updated by delta-rule learning, and uses its readout to generate low-rank corrections to the backbone's attention computation during generation. With only an $8\times8$ online memory state, $δ$-mem improves the average score to $1.10\times$ that of the frozen backbone and $1.15\times$ that of the strongest non-$δ$-mem memory baseline. It achieves larger gains on memory-heavy benchmarks, reaching $1.31\times$ on MemoryAgentBench and $1.20\times$ on LoCoMo, while largely preserving general capabilities. These results show that effective memory can be realized through a compact online state directly coupled with attention computation, without full fine-tuning, backbone replacement, or explicit context extension.

1 Introduction

The introduction argues that long-term LLM use needs compact, dynamically maintained historical memory that can steer a frozen backbone, since longer contexts are costly and may not ensure effective use of history. It presents δ-mem as an online associative state coupled directly to attention and reports gains over the frozen backbone and strongest non-δ-mem memory baseline.

  • Standard attention has quadratic cost with context length, and simply enlarging the context window may still leave additional information difficult to use.Source: §1 · p. 1
  • Textual, outside-channel, and parametric memory methods respectively face context and compaction limits, integration overhead and misalignment, or limited adaptation to evolving information.Source: §1 · p. 2
  • δ-mem updates a fixed-size associative-memory state online with delta-rule learning, then uses current-input queries to produce low-rank attention corrections while keeping the full-attention backbone frozen.Source: §1 · p. 2
  • Final average score improves by 1.10× over the frozen backbone and 1.15× over the strongest non-δ-mem memory baseline using an 8 × 8 state.Source: §1 · p. 2
  • MemoryAgentBench improves over 1.31× and LoCoMo over 1.20×; TTL nearly doubles from 26.14 to 50.50.Source: §1 · p. 2
  • An 8 × 8 OSAM retains useful historical signals and can recover context-relevant information after explicit history is removed.Source: §1 · p. 2

2 Preliminaries

δ-mem adds an online associative-memory state to a frozen Transformer: it reads compressed history to steer attention, then writes current key–value information through gated delta-rule updates.

2 Preliminaries

  • δ-mem uses a fixed-size state matrix to store key–value associations, updating it with residual delta-rule writes and a gate that controls retention.Well-learned associations cause negligible updates, while prediction discrepancies correct the state.Source: §2 · p. 3

3.1 Memory Projections

  • Hidden states are projected into low-dimensional memory query, key, and value vectors; query/key normalization mitigates scale drift, while dimension-wise gates enable selective retention and writing.The memory projections operate in a low-dimensional associative space.Source: §3.1 · p. 3

3.2 Reading from Online State of Associative Memory

  • The read vector queries the fixed-size state, so reading cost is independent of history length; it supplies continuous steering signals rather than text segments or added context tokens.Unlike standard attention over explicit-context keys, this read retrieves signals directly from compressed associative memory.Source: §3.2 · p. 4

3.3 Steering Attention through Low-Rank Corrections

  • Read signals generate dynamic query-side and output-side low-rank corrections: the former adjusts the frozen query, while the latter is added after attention.Because the read signal depends on the online state, fixed correction parameters can steer attention differently across histories.Source: §3.3 · pp. 4–5

3.4 Writing into Online State of Associative Memory

  • The gated delta write retains prior state, removes the old prediction along the current key direction, and writes the new value there, correcting errors rather than accumulating outer products indiscriminately.Dimension-wise gates independently control retention and residual writing, supporting stable historical information alongside adaptation to new inputs.Source: §3.4 · p. 5

3.5 Writing Granularity of Online State

  • TSW writes every token, SSW averages each segment to reduce redundant writes but loses detail, and MSW uses parallel sub-states to reduce interference among memory types.Token-level writes preserve fine-grained information but are more susceptible to formatting symbols, repetition, and short-term noise.Source: §3.5 · pp. 5–6

3.6 Training Objective

  • Training uses autoregressive cross-entropy on response tokens after context tokens are written to state; the frozen backbone receives only the query and response, while δ-mem parameters train.The backbone parameters remain frozen during this objective.Source: §3.6 · p. 6

4 Experiments

The experiments compare δ-mem with representative memory mechanisms and assess memory effectiveness alongside general reasoning and instruction-following across several benchmarks.

4.1 Experimental Setup

  • HotpotQA, GPQA-Diamond, and IFEval assess general tasks, while LoCoMo and MemoryAgentBench evaluate memory across extended interaction histories.LoCoMo excludes its adversarial question category; MemoryAgentBench measures retention, retrieval, and utilization.Source: §4.1 · p. 6
  • Comparisons use a shared Qwen3-4B-Instruct backbone and cover textual, parametric, and outside-channel memory baselines.The baselines are BM25 RAG, LLMLingua-2, MemoryBank, Context2LoRA, MemGen, and MLP Memory; rank-8 parameter counts are reported in Appendix C.Source: §4.1 · p. 6
  • The study selects Qwen3-8B, Qwen3-4B-Instruct, and SmolLM3-3B backbones; further training and evaluation configurations appear in Appendix A.The baseline comparisons use the shared Qwen3-4B-Instruct backbone.Source: §4.1 · p. 6, Implementation Details · p. 7

Implementation Details

δ-mem improves benchmark performance across memory mechanisms and backbone models, with results varying by benchmark, writing strategy, and model capacity.

4.2 Main Results across Memory Mechanisms

  • 51.66% average: TSW δ-mem outperforms Qwen3-4B-Instruct (46.79%) and Context2LoRA (44.90%) in the main benchmark comparison.On MemoryAgentBench, δ-mem raises the average from 29.54% to 38.85%; on LoCoMo, MSW scores 49.12%, while TSW lifts HotpotQA EM/F1 from 42.35%/56.00% to 49.41%/63.66%.Source: Table 1 · p. 7, §4.2 · p. 7
  • Textual baselines show inconsistent gains, Context2LoRA generalizes less robustly, and MLP Memory lacks sequential state accumulation and explicit long-range dependency modeling.The passage suggests that online-state memory provides more robust performance across general and memory-heavy evaluations.Source: §4.2 · p. 7

4.3 Main Results Across Different Backbone Models

  • δ-mem raises average scores from 46.79% to 51.66% on Qwen3-4B-Instruct, 47.20% to 50.86% on Qwen3-8B, and 26.08% to 36.96% on SmolLM3-3B.SSW secures the top average on Qwen3-8B, while MSW drives the larger gain on SmolLM3-3B; the passage attributes these differences to capacity-dependent writing-strategy effects.Source: §4.3 · p. 7, Table 2 · p. 8

5 Ablative Study

The ablations evaluate context recovery on HotpotQA and LoCoMo, test attention-branch choices, and compare insertion depth; they favor qo for efficiency and all-layer insertion for depth coverage.

5.2 Heads Ablation

  • 48.05% average score is achieved by qkvo, while the output branch alone reaches 47.05%; qo is the default because qkvo’s marginal gain does not justify its extra parameter overhead.The study finds that combining branches improves performance, with query-side and output-side corrections providing an effective interface for memory injection.Source: §5.2 · p. 8, Table 3 · p. 9

5.3 Insertion Depth Ablation

  • 47.97% average score is achieved by applying memory correction to all layers, alongside 49.41%/63.66% HotpotQA EM/F1 and a 46.53% LoCoMo average.The authors suggest that memory signals are most effective when they can influence representations across the backbone’s full depth.Source: §5.3 · p. 9, Table 4 · p. 9
  • 46.66% average score makes the middle-layer configuration the strongest partial-layer variant, outperforming both front-layer and back-layer configurations.The authors attribute this to intermediate layers balancing semantic abstraction and task-specific computation; front-layer injections are too local, while back-layer injections leave little depth for propagation.Source: §5.3 · p. 9, Table 4 · p. 9

6 Related Work

Related work organizes memory as textual stores, latent external memory, or parameter edits. δ-mem instead uses a compact online state to generate runtime low-rank attention corrections.

Textual Memory Mechanisms

  • Textual-memory systems store entries or summaries and retrieve them through context or RAG, but depend on compression fidelity, retrieval quality, and context budget.Early retrieval-augmented systems support knowledge-intensive generation, while agent-oriented methods organize interaction history through logging, summarization, and reflection.Source: §6 · pp. 9–10
  • δ-mem avoids reinserting compressed history as text, using a compact online state to steer the frozen Transformer through low-rank attention corrections.This separates memory maintenance from prompt-level reinsertion.Source: §6 · p. 10

Outside-Channel Memory Mechanisms

  • Latent-memory methods preserve internal representations, but separate retrieval or reader pathways can add overhead, fusion complexity, and mismatch with current representations.Memorizing Transformers store past representations as non-differentiable key-value memories retrieved with approximate kNN, while LongMem reads from an external bank through an adaptive residual side network.Source: §6 · p. 10

Parametric Memory Mechanisms

  • Parametric methods such as Prefix-Tuning, LoRA, ROME, and MEMIT add or edit parameters, but typically lack continuously evolving, history-conditioned memory at interaction-level granularity.δ-mem shares LoRA’s low-rank interface but generates attention corrections from a compact online state at runtime, rather than using static low-rank updates.Source: §6 · p. 10

7 Conclusion

δ-mem adds compact online associative memory to a frozen full-attention backbone, improving memory-heavy benchmark performance while largely preserving general capabilities.

7 Conclusion

  • δ-mem improves performance on memory-heavy benchmarks while largely preserving the frozen backbone’s general capabilities.The conclusion reports that an 8 × 8 online state can recover useful historical information after explicit context is removed.Source: §7 · p. 10
  • δ-mem compresses past information into a fixed-size, dynamically updated online state whose readout generates low-rank corrections to attention components.This lets memory participate directly in forward computation without full fine-tuning or replacing the backbone architecture.Source: §7 · p. 10
  • The conclusion suggests compact online states can provide a scalable, efficient test-time memory interface without extending explicit context or using heavy external retrieval modules.Source: §7 · p. 10

Appendix · A Implementation Details

The appendix specifies a one-epoch QASPER training setup for δ-mem and describes the evaluation metrics and sample-weighted aggregation used across benchmarks.

A Implementation Details

  • Models train for one epoch on QASPER’s 2,219-sample split, with a maximum backbone sequence length of 512 and a memory write budget of 8,192 tokens.The split’s maximum sequence length is 8,269 tokens.Source: A Implementation Details · p. 14
  • Unless otherwise specified, δ-mem uses r = 8 and α = 16 on the query and output branches; MSW uses 4 states.Source: A Implementation Details · p. 14
  • Training uses 8 A800 GPUs, bfloat16 precision, DeepSpeed ZeRO-2, and fused AdamW.Source: A Implementation Details · p. 14
  • Optimization uses a 2 × 10−4 peak learning rate with cosine decay and 0.1 warmup ratio; batch size is 1 per device with 4 accumulation steps, for global batch size 32.The random seed is 42.Source: A Implementation Details · p. 14
  • Evaluation follows official prompts and decoding settings, reporting prompt-level strict accuracy for IFEval, EM and F1 for HotpotQA, accuracy for GPQA, and F1 for LoCoMo.Source: A Implementation Details · p. 14
  • MemoryAgentBench’s evaluation categories, datasets, and metrics are summarized in Table 5, and its final score is the sample-weighted average across datasets.Source: A Implementation Details · p. 14, Table 5 · p. 14

B Inference Efficiency and Memory Use

δ-mem keeps GPU memory use nearly at the level of Vanilla and Context2LoRA as prompts grow to 32K, while decoding more slowly than those methods but faster and more stably than MemGen. The results show a lightweight computational footprint alongside memory capability.

B Inference Efficiency and Memory Use

C Parameter Overhead

δ-mem adds little trainable-parameter overhead: its SSW and TSW variants use 0.12% of the backbone’s parameters, while MSW uses 0.48%, compared with much larger auxiliary-memory baselines.

C Parameter Overhead

  • The MSW variant, which maintains multiple memory states, requires 19.47M trainable parameters, or 0.48% of the backbone.Despite using multiple memory states, MSW remains a small fraction of backbone parameter scale.Source: C Parameter Overhead · p. 15
  • These parameter counts show that δ-mem provides online memory augmentation with substantially lower overhead than larger auxiliary-memory modules.The paper characterizes δ-mem as a lightweight alternative to those modules.Source: C Parameter Overhead · p. 15
  • SSW and TSW each add 4.87M trainable parameters (0.12% of the backbone), versus 46.20M for MemGen and 3078.00M for MLP Memory (76.40% of the backbone).Figure 4 compares trainable parameter counts across memory-augmented methods.Source: Figure 4 · p. 15, C Parameter Overhead · p. 15

Related papers

Loading 2605.12357v1…