PaperlayerView exact revision on arXiv

Source-linked AI summary

Constant-Memory Recall: Learned Associations in a Fixed Matrix State

Samuel Larson

arXiv:2610.00232v1cs.LG

TL;DR

Fixed-size recurrent memory can bound inference storage, but successful recall depends on the task and training. This paper tests a small DeltaNet variant on 32 new key–value pairings per sequence and finds near-perfect retrieval with 32 KiB of matrix state, while unsuccessful baselines leave memory-efficiency comparisons unresolved.

  • Problem

    Fixed-size recurrent memory can bound inference storage, but retaining facts longer differs from storing more facts or following association chains.

    Source: §1 · p. 1

  • Method

    The study trains a small DeltaNet variant to retrieve 32 new key–value pairings per synthetic sequence and compares it with vector and Transformer baselines.

    Source: §1 · p. 1, §2.1 · p. 1

  • Results

    0.9995 mean accuracy: the matrix model retrieves among episode values, while the vector and Transformer baselines remain near chance.

    Source: §3.1 · pp. 2–3

  • Takeaways & Limitations

    The experiments document retrieval of 32 pairings from a fixed 32 KiB matrix state, not a general capacity law or memory-efficiency advantage over attention.

    Source: §6 · p. 5

    The experiments use one small architecture and synthetic episodes, and failed baselines prevent an architectural-efficiency claim or conclusions about pretrained models and natural text.

    Source: §5 · p. 5

Abstract

from arXiv · show

Fixed-size recurrent memory limits storage growth during inference, but successful recall depends on the task and training. We study a small DeltaNet variant with fixed token-specific key biases, trained to remember 32 new key-value pairings per sequence. With 32 KiB of recurrent matrix state, it achieves 99.95% mean accuracy across three training seeds when choosing among the sequence's values. Recall remains near perfect when filler extends the pre-query context to 1,798 tokens without adding pairings. Zeroing the first memory block removes this recall. An exploratory 48-pair test remains near chance after one quarter of the primary training budget and does not locate a capacity limit. Parameter-matched vector and Transformer baselines remain near chance, including the Transformer after additional training searches. This unresolved baseline failure prevents a memory-efficiency comparison.

1 Introduction

Transformer caches grow with context, whereas fast-weight models update fixed-size memory; retaining facts longer is distinct from storing more facts or following association chains. This study examines retrieval and filler retention in a small DeltaNet variant, while unsuccessful controls leave memory efficiency unresolved.

1 Introduction

  • Fixed-size fast-weight memory can bound inference memory, but longer retention differs from storing more facts or following association chains.The introduction contrasts this with a Transformer key–value cache that grows with context length.Source: §1 · p. 1
  • The study tests retrieval and filler retention in one small DeltaNet variant, with state interventions, linear probes, and an exploratory higher-load check.Learned associative recall with fast weights and multi-query recall with DeltaNet were previously established (Ba et al., 2016; Schlag et al., 2021; Yang et al., 2024b).Source: §1 · p. 1
  • Approximately 14-million-parameter models trained on the same synthetic episodes show recall learning only in the matrix model, so the comparison cannot establish memory efficiency over attention.The vector control also has less memory and a different write operation; the Transformer fails even with its full cache, so the difference is not attributed to matrix structure alone.Source: §1 · p. 1

2 Experimental setup

The experiments use synthetic episodes with 32 cyclic key–value pairings and score retrieval among episode values. They compare a two-block matrix model with vector and Transformer controls, then evaluate training-seed variation under a preregistered primary comparison.

2.1 Recall task and scoring

  • Each episode links 32 distinct entity tokens in a random cycle; a six-token query asks for a key’s successor, scored among the episode’s values.Episode-restricted chance accuracy is 1/32 = 0.03125, and the preregistered recall threshold is 3/32 = 0.09375; this is not full-vocabulary generation.Source: §2.1 · pp. 1–2

2.2 Models and training setup

  • The matrix model uses two 64 × 64 DeltaNet states; the parameter-matched vector control has a 512-byte state and a different write operation.The matrix states occupy 32,768 bytes independent of context length, excluding model weights, fixed bias tables, convolution history, and temporary activations; the vector control changes both memory size and write operation.Source: §2.2 · p. 2

2.3 Analysis

  • Three seeds are evaluated on 4,096 held-out queries, and the primary comparison requires matrix recall above threshold plus a paired-seed 95% interval exceeding a 0.30 advantage over the vector model.The interval measures training-seed variation on a fixed query set, not independent variation across queries; cache limits count as an efficiency test only if the full-cache Transformer first learns recall.Source: §2.3 · p. 2

3 Results

The matrix model learns single-step recall for 32 pairings and retains it through longer filler contexts. State interventions, readout probes, and a shorter higher-load test qualify what these results establish.

3.1 Single-step recall

  • 0.9995 mean accuracy and a 0.9656 paired accuracy advantage over the vector recurrence (95% interval 0.9582–0.9729) accompany near-chance parameter-matched baselines.The matrix model learns in all three seeds; the additional Transformer search found no evaluated seed above the recall threshold, with its cause unresolved.Source: §3.1 · pp. 2–3, Table 1 · p. 3

3.2 Recall over longer sequences

  • At 454, 902, and 1,798 pre-query tokens, the matrix model scores 0.9995, 0.9983, and 0.9990 by seed using the same 32,768-byte state.The contexts add filler but no pairings; these tests show retention through filler, not resistance to new interfering associations or advantage over a successful attention baseline.Source: Figure 1 · p. 3, §3.2 · pp. 3–4

3.3 State interventions and readout probes

  • Zeroing S0 reduces accuracy below the 0.094 recall threshold, while zeroing S1 leaves it nearly unchanged.Across seeds, removing S0 yields 0.0339, 0.0012, and 0.0002 accuracy; removing S1 yields 0.9995, 0.9949, and 0.9990.Source: Figure 2 · p. 4, §3.3 · p. 4
  • No tested raw-state/query ridge probe recovers target value vectors above cosine 0.9; a hidden-state probe reaches mean cosine 0.894 and threshold pass fraction 0.674.The hidden-state result uses a separate seed-0 calibration checkpoint, and regularization was selected on the same evaluation set, so the findings are descriptive.Source: §3.3 · p. 4

3.4 Exploratory higher-load test

  • With 48 pairings, all scores are near chance (0.021), but the check uses one quarter of the primary training budget and establishes neither a capacity boundary nor failure at matched training.The 48-pair check uses the same state width.Source: Table 2 · p. 4, §3.4 · p. 5

4 Related work

Prior work establishes fast-weight associative recall and connects the delta-rule update to linear attention. This paper distinguishes its terminal-query setting from prior multi-query recall and notes limitations in its baseline comparisons.

Fast-weight memory

  • Fast weights have been used for temporary associations (Ba et al., 2016), and Schlag et al. (2021) connect them to linear attention, introduce the delta-rule update, and test retrieval, overwriting, and memory load.DeltaNet achieved perfect accuracy on its hardest tested multi-query recall setting (Yang et al., 2024b), while Gated DeltaNet distinguishes repeated filler from interfering natural text (Yang et al., 2024a).Source: §4 · p. 5

Recall comparisons

  • Zoology (Arora et al., 2023) finds attention stronger than tested recurrent alternatives, while Okpekpe & Orvieto (2025) show optimization and model dimensions affect recall comparisons.This paper uses a terminal query and restricted candidate set rather than multi-query recall; its vector control also lacks memory matching and does not test key-dependent binding alternatives.Source: §4 · p. 5

Memory comparisons

  • Cache-reduction methods (Xiao et al., 2023; Zhang et al., 2023) assess how much cache preserves a working model’s behavior, but the unsuccessful full-cache baseline here prevents that comparison.Capacity results for associations stored in trained parameters (Nichani et al., 2025) concern a different setting from fresh pairings in recurrent state.Source: §4 · p. 5

5 Limitations

The study tests one small fast-weight architecture trained from scratch on synthetic episodes, so its findings have limited scope. Its tests do not establish performance on pretrained models, natural text, or deployment-length contexts.

Scope and evaluation

  • The study does not establish performance on pretrained models, natural text, or deployment-length contexts, and its longer sequences add filler rather than new bindings.The primary result rests on three training seeds and one fixed query set; the higher-load check has a shorter training budget, and the hidden-hop test cannot measure multi-hop ability.Source: §5 · p. 5

6 Conclusion

The small DeltaNet retrieves 32 pairings from fixed 32 KiB matrix state and retains them through tested filler sequences, but the experiments do not establish broader capacity or efficiency claims.

6 Conclusion

  • The small DeltaNet retrieves 32 pairings from fixed 32 KiB matrix state and retains them through tested filler sequences; the experiments establish neither a general capacity law nor an efficiency advantage over attention.They also do not establish extension to higher loads, specified hop counts, or natural text.Source: §6 · p. 5

A Recall across cache sizes

  • Table 3 reports mean recall across three seeds by pre-query context length, with matrix state fixed at 32,768 bytes and Transformer cache caps set as multiples of that size.Chance is 0.031 and the recall threshold is 0.094.Source: Table 3 · p. 7
  • The M = 1 eight-token cap was excluded because it falls below the 13-token protocol floor, and its descriptive scores were 0.021–0.030.Source: Table 3 · p. 7

B Linear readout results

The readout analysis distinguishes three query pathways and reports seed-specific pre-LM-head probe pass fractions; Table 4 characterizes ridge-probe recovery as descriptive.

B Linear readout results

  • Table 4 reports descriptive ridge-probe recovery of target value vectors, with “rf@0.9” counting cases above cosine 0.9 and “gap” subtracting a shuffled-fit control.Only the matrix model’s pre-LM-head hidden state supports substantial recovery.Source: Table 4 · p. 7
  • “Shallow query” uses the second block’s query pathway on query-token embeddings, while “bind-path query” uses the first block’s pathway.Source: B Linear readout results · p. 7
  • Across the three primary training seeds, pre-LM-head probe pass fractions are 0.686, 0.771, and 0.951.Source: B Linear readout results · p. 7

C Training loss

After 20,000 steps, the matrix model has lower training loss than the vector recurrence and Transformer, while the baselines’ decreasing loss does not produce recall.

C Training loss

  • After 20,000 steps, training loss is 1.38 for the matrix model, 7.69 for the vector recurrence, and 7.51 for the Transformer.Source: Figure 3 · p. 8
  • The vector recurrence and Transformer losses decrease without producing recall.Source: Figure 3 · p. 8
  • The plotted runs use the shared settings of Section 2 and report seed 0.Source: Figure 3 · p. 8

D Transformer search and reproducibility

The Transformer search evaluates multiple depths, widths, learning rates, and training lengths, but no evaluated seed clears the recall threshold; the figure script verifies archived artifacts.

D Transformer search and reproducibility

E Diagnostic with an unspecified hop count

The diagnostic selects targets one or two steps along the episode cycle without specifying the hop count to the model. Its recorded accuracies do not establish multi-hop capability or failure to learn it.

E Diagnostic with an unspecified hop count

Related papers

Loading 2610.00232v1…