PaperlayerView exact revision on arXiv

Source-linked AI summary

Kathleen Remembers: Length-Invariant One-Shot Recall Without Attention

George Fountzoulas

arXiv:2608.30376v1cs.CLcs.LG

TL;DR

Fading recurrent states cannot exactly recall a one-time key–value pair far in the past, so the paper adds a fixed-size holographic notebook to an attention-free byte trunk. The notebook reaches 80–82% one-shot recall at 4× training length, improves distant rare-word repetition, and derives selective erasure and provenance from linear superposition. Its scope remains exact recall: attention generalizes on natural-text repetition, structured unmarked needles remain difficult, and larger compute regimes are unexplored.

  • Problem

    Fading recurrent states lack exact recall of a one-time key–value pair far in the past, motivating a memory layer that preserves the family’s linear-cost constraints.

  • Method

    The paper attaches a 25K-parameter fixed-key holographic associative store with a local write gate, self-gating raw read, and write-triggered forgetting to the trunk logits.

  • Results

    Across controlled recall, algebraic, natural-text, and scaling experiments, the notebook sustains 80–82% recall at 4× training length, enables 100% provenance, improves distant repetition, and gains monotonically with data scale.

  • Takeaways & Limitations

    The notebook supplies length-invariant exact recall and arithmetic unlearning and attribution while preserving the paper’s attention-free, constant-state design constraints.

  • Takeaways & Limitations

    Unmarked needle detection reaches only 53–64%, structured domains without markers need a future hybrid gate, and regimes beyond ∼0.5M parameters remain unexplored.

Abstract

from arXiv · show

Recurrent, attention-free sequence models share a structural weakness: a fading state cannot perform exact recall of something seen once, far in the past. We add to the Kathleen trunk a second memory layer -- a "notebook": a fixed-key holographic (HRR) associative store with a learned local write gate, a self-gating raw read, and write-triggered forgetting -- 25K parameters that attach to the logits of any trunk. (1) Mechanism: on a controlled needle-in-haystack task the notebook reaches 80-82% one-shot recall at 4x the training length, where the bare trunk scores ~4% and a parameter-matched attention head scores 100% inside its training length and 0% beyond it. Addressing is length-invariant by construction; the untrained memory alone recalls at 90% accuracy identically at 512, 2048 and 4096 bytes. Because the store is a linear superposition, two capabilities follow from arithmetic alone: selective unlearning (one subtraction erases one fact to chance, retained facts unharmed) and per-token attribution (counterfactual erasure names the source fact of every correct byte, 100% provenance). (2) Real text: on WikiText-2 bytes the notebook improves prediction of repeated rare words by +0.15-0.27 bits/byte, the gain growing with the distance between mentions and holding zero-shot at 4x training length; write-triggered forgetting eliminates memory pollution at 8x length (first-mention cost +0.33 -> -0.004). (3) Scope and scale: a parameter-matched attention head does generalize on natural-text repetition, so the notebook's claim is exact recall at O(L); on a WikiText-103 ladder (8 to 512 MB) the zero-shot repeat gain rises monotonically. All experiments are pre-registered, seeds reported, and reproducible on a single free-tier GPU.

1 Introduction

The paper adds a fixed-size notebook to an attention-free byte model, targeting exact long-range recall while preserving linear cost, constant state, and byte-native processing. It reaches substantial one-shot recall beyond training length, supports algebraic erasure and provenance, improves distant natural-text repetition, and has explicit scope boundaries.

  • Contribution: 25K parameters attach a fixed-key holographic notebook to any trunk while preserving O(L), constant state, and byte-native processing.The notebook uses fixed content keys, a local write gate, self-gating reads, and write-triggered forgetting.
  • Mechanism result: 80–82% one-shot recall at 4× training length exceeds the bare trunk’s ∼4% and contrasts with a parameter-matched attention head’s 0%.The untrained memory path alone reaches 90% at 512, 2048, and 4096 bytes.
  • Algebraic consequences: Linear-superposition arithmetic enables selective unlearning to chance without harming retained facts and 100% per-token provenance without retraining.Both capabilities use subtraction and are length-invariant.
  • Real-text result: The notebook improves repeated rare-word prediction at distance with zero overall cost, while its unsupervised write gate discovers content-word salience.The improvement is reported on natural text where the recurrent state has faded.
  • Boundaries: Attention generalizes on loose natural-text repetition, so the notebook’s claim is exact recall rather than universal repetition superiority.The notebook is absorbed by a word-level model because words do not require exact answers.
  • Scaling result: +0.06 →+0.16 bits/byte rises monotonically across the 8 →512 MB ladder, with zero-to-negative overall cost throughout.The top rung uses two seeds; the compute-optimal regime beyond ∼0.5M parameters remains unexplored.

2 The two-layer memory

The two-layer memory combines a fading recurrent timing layer with a fixed-key holographic associative store. Its design choices isolate content-addressed recall from length-dependent trunk behavior while using local writes, raw-magnitude reads, and content-tied forgetting.

  • 2.1 The trunk (recap): The unchanged trunk uses leaky integrators across fast, medium, and slow half-life regimes, storing a fading mixture rather than retrievable items.This layer provides timing information but not exact item recall.
  • 2.2 The notebook: The notebook is one 2048-dimensional holographic store per model, using circular-convolution binding and correlation-based unbinding.The scan costs O(d_hrr log d_hrr) per position and is parallelizable.
  • 2.3 Why each piece is what it is: Fixed content keys make addresses depend on text rather than position, providing length generalization by construction.The keys are random unit vectors composed from the last seven bytes with fixed role vectors.
  • 2.3 Why each piece is what it is: A local write gate avoids length-dependent policies because it sees only a 16-byte causal window, not sequence length, position, or global state.Its window must span the needle: an 8-byte gate never learns the correct write rule.
  • 2.3 Why each piece is what it is: Raw read magnitudes provide hit/no-hit gating, avoiding normalized empty-memory noise that can mute genuine retrievals.The normalized-read oracle arm reached only 55–65% in the cited failure analysis.
  • 2.3 Why each piece is what it is: Write-triggered forgetting ties decay to content instead of elapsed time, preventing a clock from reintroducing length dependence.With unitary keys, targeted delta-rule erasure is algebraically equivalent to global decay.
  • 2.3 Why each piece is what it is: Variable-length trunk training is required because fixed-length readout weights drift at 4× length and can override a correct memory.The memory-only counterfactual reached 90% at 2048 while the full model scored 0.5% before variable-length training.

3 The needle laboratory

The needle laboratory isolates why exact one-shot recall fails beyond training length and shows that fixed-content associative memory can remain length-invariant when other length leaks are controlled.

  • Headline arms: The attention head is perfect at its 512-byte training length but collapses to zero at 2048 bytes, reproducing a zero-shot length cliff.Counterfactual read-outs attribute recall to the memory path rather than the trunk.
  • Forensic path: Fixed content keys remove the learning cliff but do not solve zero-shot recall; increasing HRR dimension improves training while zero-shot performance remains near zero.At d=2048, training reaches 96–97%, while zero-shot remains approximately 0%.
  • Forensic path: Variable-length training from 128–512 cures the trunk drift, yielding 80–82% learned-gate recall that remains flat across distance bins at 4× length.The oracle reaches 91→92% without degradation at 4×.
  • Forensic path: Three independent leaks are isolated: local-gate width, normalized-read noise amplification, and trained-trunk logit drift at unseen lengths.Oracle and counterfactual tests show the associative memory itself was length-invariant.

4 Consequences of the algebra: unlearning and attribution

The notebook’s linear superposition supports selective unlearning and per-token provenance through subtraction, without retraining or architectural changes.

  • Selective unlearning: One vector subtraction erases a selected fact at cost O(span), while leaving the other stored facts untouched.The fact’s bound vectors are recomputed from its span because keys are fixed and content-derived.
  • Selective unlearning: The erased fact drops to chance while four retained facts remain unharmed, and memory-only readout confirms the information is absent from the store.Table 2 defines chance as 3.8%.
  • Per-token attribution: Counterfactual erasure identifies the source fact for each emitted byte by measuring which fact’s removal causes the largest logit drop.The same subtraction underlies both attribution and unlearning.
  • Per-token attribution: Provenance reaches 100% on correct value bytes, while 96.4/95.2% are memory-decisive at the two tested lengths.Notebook interference with ordinary filler prediction is +0.15/+0.03 percentage points, effectively none.

5 Real text: the repeat probe

On WikiText-2 bytes, the notebook improves prediction specifically for repeated rare words, with gains increasing over longer distances and remaining effective zero-shot. Write-triggered forgetting removes the long-context cost caused by memory pollution.

  • 5.2 Result: +0.19/+0.15 bits/byte at 512 and +0.27/+0.24 at 2048: extra repeat gain grows as mention distance increases.The gains are measured across two seeds and increase where the fading recurrent state has lost information.
  • 5.2 Result: The write gate discovers content-word salience without supervision, assigning β=0.28 to repeated-word bytes versus 0.19 elsewhere.The gate is trained only through the language-modeling loss.
  • 5.3 The one failure, and forgetting: +0.33 first-mention bpb cost at 4096 is reduced to −0.004 by write-triggered forgetting.Forgetting ties decay to write mass, preventing junk accumulation and superposition noise at 8× training length.
  • 5.3 The one failure, and forgetting: +0.09/+0.16/+0.24 gain remains positive across every distance bin at 4096 after forgetting.Near-bin gains shrink because the eraser also removes some useful memory.
  • 5.4 Scope: what attention does here: +0.83 · · · +1.04 repeat gains for the parameter-matched attention twin hold at 2048 zero-shot on natural text.The paper therefore scopes its claim to exact recall of once-seen material at O(L) and constant state.

6 The wrong habitat: a word-level negative result

At word level, the notebook’s repeat benefit is largely absorbed rather than reproduced. The paper attributes this boundary to the rarity of exact-answer continuation questions in word sequences.

  • 6 The wrong habitat: a word-level negative result: +0.07/−0.06 extra repeat gain at training length and +0.08/+0.13 at 4× zero-shot show limited word-level benefit.The same recipe uses content keys per word, 3-gram bundles, and an 8-word gate.
  • 6 The wrong habitat: a word-level negative result: No gate selectivity appears at word level, with β flat at 0.16.Overall cost remains zero, but the memory provides little additional repeat gain.
  • 6 The wrong habitat: a word-level negative result: Exact-recall memory is useful when the prediction question has an exact answer, as byte continuations can after a partial word.Word-level contexts around repeated mentions usually differ, so context-derived keys match nothing.

7 Scaling: the ladder

Across a WikiText-103 data ladder, the notebook’s zero-shot repeat value rises monotonically while overall cost stays zero-to-negative. The bare trunk does not develop long-range exact recall through data scaling alone.

  • 7 Scaling: the ladder: Zero-to-negative overall cost holds on every data rung.The ladder comparison is budget-matched across notebook and trunk arms.
  • 7 Scaling: the ladder: −0.04 · · · −0.05 bare-trunk repeat gain persists on every rung, so long-range exact recall does not emerge from data scale in this family.The parameter-matched attention twin reaches +0.79 extra repeat gain on the 512 MB rung for natural-text repetition.
  • 7 Scaling: the ladder: At 64× data, the notebook reaches a 2.4× gain at 8× training length.Figure 3 places this result on a logarithmic training-data axis with a two-seed error bar at the top rung.

8 Related work

The notebook builds on fading-state recurrent models and prior associative-memory approaches while changing how memory is addressed. Its fixed content-derived addressing sidesteps attention’s length cliff at O(L), and its role is complementary to decode-time retrieval.

  • Fading-state recurrent models: S4, Mamba, RWKV, and Griffin are O(L) fading-state trunks with a documented weakness in exact long-range recall.The Kathleen trunk is architecturally kin to these models and inherits the same diagnosis.
  • Associative memory: Prior linear-attention and fast-weight programmers include delta-rule notebooks whose length-generalization failures motivate the content-key redesign.The paper presents this redesign as its departure point from that family.
  • Explicit memory modules: Unlike explicit memory modules, the notebook uses fixed, content-derived, length-blind addressing with a 25K-parameter budget.The comparison includes NTM/DNC, Memorizing Transformers, Titans, and other memory layers.
  • Transformer length generalization: RoPE extensions, YaRN, and ALiBi patch attention’s length cliff, whereas the notebook sidesteps it at O(L).The distinction is between engineering attention for longer contexts and using a separate fixed-state memory path.
  • Retrieval-augmented decoding: The notebook is a train-time, in-context counterpart to retrieval-augmented decoding.Both support memory where answers are exact, but they operate at different pipeline stages.

9 Limitations •

The paper identifies unresolved boundaries in salience detection, attention comparisons, forgetting, word-level transfer, and compute scale.

  • The compute-optimal regime beyond ∼0.5M parameters remains unexplored, with the ladder’s monotonic trend based on four points and a two-seed error bar at the top.
  • Un-marked needle detection reaches only 53–64%, while the wide local gate alone scores 4%; hybrid local+context gating remains future work.Structured domains without markers may therefore require a different salience mechanism.
  • The attention comparison uses one budget-matched causal head on the same trunk, not a tuned modern transformer with rotary extrapolation tricks.
  • Forgetting trades away some near-distance gain, and unitary-key algebra makes targeted erasure unavailable within the superposition design.Alternative store geometries, such as slot-based memories, could reopen this trade-off.
  • At word level, the notebook is largely absorbed, making word-level transfer a reported boundary of the method rather than a resolved result.

10 Conclusion and the road ahead

The notebook addresses the fading state’s exact-recall weakness while preserving the series’ O(L), constant-state, and byte-native constraints. Its algebraic memory also supports guaranteed fact erasure and source attribution, while future work targets scale, salience, and raw-stream applications.

  • A 25K-parameter notebook adds fixed holographic content keys, a local write gate, self-gating reads, and write-triggered forgetting to an attention-free trunk.
  • The notebook delivers one-shot recall beyond the training length, improves repeated rare-word prediction, and scales its zero-shot repeat gain with data.
  • Algebraic storage enables chance-level erasure of one fact without collateral damage and 100% per-token provenance through counterfactual erasure.
  • Future work includes a compute-optimal scale study, hybrid gating for structure-free domains, and evaluation on audio and sensor streams.
  • The disclosed memory mechanism is the subject of U.S. Provisional Patent Application No. 64/140,260, filed August 25, 2026.

A Recipe card (exact hyperparameters)

The recipe specifies the trunk, notebook, optimization, evaluation, and reproducibility settings, while Table 4 records the pre-registered experiment registry.

  • The trunk uses d=96 for the needle rig and d=128 for the ladder, with 2–3 blocks, multi-scale reverberation, causal convolution k=7, and no positional table.
  • The notebook uses d_hrr=2048, fixed normalized content keys, unitary role keys, and a learned gate built from Embedding, causal Conv1d, GELU, and Linear layers.
  • Training uses AdamW with learning rates 3e−3 for the rig and 2e−3 for the ladder, weight decay 0.01, gradient clipping 1.0, and variable-length batches.
  • Evaluation reports teacher-forced value-byte accuracy for needle tasks and bits/byte with repeat masks for text, using seeds 42/43.
  • Table 4 registers every experiment and its pre-registered verdict, with accompanying JSON reports.
Loading 2608.30376v1…