PaperlayerView exact revision on arXiv

Source-linked AI summary

How Linear Attention Remembers

Kichang Lee, JaeYeon Park, Songkuk Kim, JeongGil Ko

arXiv:2609.33093v1cs.LGcs.AI

TL;DR

Linear attention reduces growing KV-cache memory by storing history in a shared fixed-size recurrent state, raising questions about how facts are retained and retrieved. The paper combines analytical decomposition with causal interventions in pretrained GLA and GDN models. It finds selective recall alongside cross-fact coupling and interference, while hybrid recall relies predominantly on full-attention KV memory.

  • Problem

    How recurrent state stores, retains, and retrieves facts remains unclear, despite the memory costs of growing KV caches in serving and resource-limited settings.

    Source: §1 · p. 1

  • Method

    The paper decomposes recurrent state by source position and uses controlled donor–recipient interventions in pretrained GLA and GDN models to trace writes, retention, and readout.

    Source: §1 · p. 2, §2.2 · p. 2, §2.3 · p. 3

  • Results

    Pretrained models show concentrated content-specific writes and query reads, selective but coupled multi-fact recall, load-related interference, and predominantly KV-supported recall in hybrids.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    Recurrent state behaves as structured shared memory rather than independent token-specific KV entries, with interference depending on memory load and overlapping writes.

    Source: §6 · p. 9

    Experiments focus on controlled associative recall, not all forms of long-context reasoning or generation, and cover a limited set of pretrained checkpoints.

    Source: §6 · p. 9

Abstract

from arXiv · show

Linear attention replaces the growing key--value (KV) cache of standard attention with a fixed-size recurrent state, substantially reducing memory growth with context length. This efficiency, however, changes how past information is stored: many tokens must share and repeatedly update the same memory. We study how this recurrent state functions as a memory system. Using an analytical decomposition together with controlled causal interventions in pretrained GLA and GDN models, we trace how recalled information is written, retained, and later accessed. We find that fact-specific information enters recurrent memory through concentrated, content-dependent writes and is later accessed through concentrated query-time read pathways. Multiple facts can remain selectively accessible within the same state, yet their internal representations exhibit cross-fact causal coupling rather than independent KV-like storage. As memory load increases, both recall and targeted editability degrade, whereas elapsed context alone has a substantially smaller effect within the tested regime. Causal interventions on subsequent writes further show that interference is shaped by their overlap with existing memory. Finally, in hybrid architectures that combine recurrent layers with full attention, the runtime memory directly supporting recall shifts predominantly to the full-attention KV state. Together, these results reveal how fixed-size recurrent memory supports selective recall despite shared storage, while exposing the interference and capacity limits that distinguish it from token-addressable KV memory.

1 INTRODUCTION

Linear attention reduces context-dependent memory growth by replacing token-indexed KV entries with a fixed-size state, but that state repeatedly accumulates information from many tokens. This leaves open how facts are written, retained, and accessed, and whether multiple facts or hybrid attention change that memory behavior.

1 INTRODUCTION

  • Growing KV caches consume accelerator memory, constrain concurrent serving, and are especially restrictive on memory-limited mobile, edge, and robotics systems.Reducing memory for expanding contexts is therefore an important efficient-model design objective.Source: §1 · p. 1
  • Linear attention replaces per-token KV representations with a fixed-size state, substantially reducing memory required for long-context inference.Unlike token-indexed KV entries, information from many past tokens is continually updated and accumulated in the same finite state.Source: §1 · p. 1
  • How recurrent memory writes, retains, and retrieves information—and how multiple facts coexist or retention degrades—remains unclear from prior KV-cache research.It is also unclear whether recurrent state remains the primary memory substrate when full attention is available.Source: §1 · p. 2
  • The paper uses analytical decomposition and causal interventions in pretrained GLA and GDN models to trace concentrated content-specific writes and query-time reads.It also studies multi-fact selectivity, cross-fact coupling, memory-load interference, and hybrid architectures.Source: §1 · p. 2

2 BACKGROUND AND ANALYTICAL FRAMEWORK

The framework treats linear-attention memory as a fixed-size recurrent state whose contributions can be decomposed by source position and whose effects depend on later transitions and query-time reads. The evaluation uses controlled associative recall and interventions across pretrained recurrent and hybrid models.

2.1 RECURRENT STATE IN LINEAR ATTENTION

  • A recurrent head derives a query and token-dependent write from each hidden representation, updates its matrix state through architecture-dependent transitions, and later reads that state with a query.The described variants include multiplicative gating and delta-rule updates; a common write is rank one.Source: §2.1 · p. 2

2.2 AN ANALYTICAL VIEW OF RECURRENT STATE MEMORY

  • The analytical decomposition tracks each position's write through subsequent transitions into the shared state, distinguishing stored contributions from their query-dependent effects on output.The decomposition is exact along a fixed forward trajectory; contributions are not explicitly token-indexed, and retention depends on subsequent transitions rather than token age alone.Source: §2.2 · pp. 2–3
  • Because the mapping from source contributions to their summed state is generally many-to-one, the framework tests selective manipulation under multiple and overlapping writes.It separates elapsed context from competing and overlapping writes in later experiments.Source: §2.2 · p. 3

2.3 MODELS AND EVALUATION PROTOCOL

  • The evaluation uses pretrained GLA and GDN models, controlled associative recall, and paired donor–recipient interventions that transfer a state, update, or output to measure prediction shifts.Recall uses four balanced candidate values; development and confirmation sources are disjoint, and intervention components are fixed using development data.Source: §2.3 · p. 3
  • Figure 1 summarizes interventions at fact writing, before-query state replacement, query-time readout, and multi-fact sharing, with single-fact interventions confined to a selected recurrent head.The selected head is chosen on development data and frozen for confirmation.Source: Figure 1 · p. 4

3 TRACING THE CAUSAL LIFECYCLE OF RECURRENT MEMORY

We trace how information is written into recurrent state, retained through later context, and accessed at query time. The experiments show content-specific writes and concentrated read pathways, while multiple facts remain selectively accessible despite cross-fact coupling in the shared state.

3.1 CONTENT-SPECIFIC WRITES PERSIST IN RECURRENT STATE

  • Blocking a selected fact-time update reduces recall by 36–79 percentage points, and replacing it with a donor update transfers the donor value in 68–90% of cases.The donor-write effect is not reproduced by equal-norm random or control-head updates; the selected write remains behaviorally effective after intervening context.Source: §3.1 · pp. 4–5
  • Figure 3 identifies the joint Hub+B intervention as non-additive relative to the sum of the separate interventions.The figure compares the selected GLA Hub head L21H3 with secondary head B, L14H3.Source: Figure 3 · p. 5

3.2 RECALL IS MEDIATED BY CONCENTRATED QUERY-TIME READ PATHWAYS

  • Donor-state swaps and donor-output patches produce comparable answer shifts, while restoring the recipient’s clean query-time output largely removes the donor effect.These interventions support a concentrated query-time read pathway mediating the effect of information in the pre-query state.Source: Figure 4 · p. 5, §3.2 · p. 5

3.3 SHARED STATE SUPPORTS SELECTIVE MULTI-FACT RECALL

  • In four-fact contexts, donor answers transfer in 61–89% of cases while only 0.4–1.4% of previously correct non-target answers break.The results show behavioral selectivity despite shared recurrent storage.Source: Figure 5 · p. 6, §3.3 · p. 6
  • Fact A’s state-difference components parallel and orthogonal to fact B’s direction exert opposing effects on B, consistent with cross-fact causal coupling.Norm-matched Gaussian controls have much smaller effects, and the complete edit has a weaker signed effect, consistent with partial cancellation.Source: Figure 5 · p. 6, §3.3 · p. 6

4 GENERALITY AND PRACTICAL CONSEQUENCES

The section tests where recall is carried in hybrid models and how competition limits shared recurrent memory. In tested hybrids, recall shifts to full-attention KV state; in recurrent models, increasing load and overlapping writes weaken recall and targeted edits.

4.1 THE MEMORY SUBSTRATE SHIFTS IN HYBRID ARCHITECTURES

  • In near-matched hybrid GDN-340M, recurrent/local replacement recovers less than 1% of the donor effect, versus over 99% for KV replacement; this pattern also holds across tested Qwen3.5 hybrid conditions.The comparison is treated as an architectural boundary, not an effect attributable solely to replacing recurrent blocks.Source: Figure 6 · p. 7, §4.1 · p. 7

4.2 ENTANGLED STATE MEMORY LIMITS SELECTIVE MANIPULATION

  • At fixed sequence length, increasing active associations from K = 1 to K = 16 reduces recall by roughly 30–40 percentage points, while extending fact-to-query distance from 128 to 1024 tokens at K = 4 causes only a modest decline.In the tested regime, this contrast suggests competition within shared state matters more than passive loss over distance.Source: §4.2 · pp. 7–8, Figure 7 · p. 7
  • Aligning a later distractor’s write key with the target direction lowers target margin by about 0.27–0.44 logits across tested loads, while orthogonalization and control-component alignment have little mean effect.The intervention indicates that overlap with the target direction can directly disrupt its contribution to shared state.Source: Figure 7 · p. 7, §4.2 · p. 8
  • As load rises, clean recall and target-edit strength decline, while collateral damage differs by model: GDN models stay below a 2% non-target break rate through K = 32, but GLA-340M exceeds the 5% criterion from low load onward.At K = 32, recall is roughly 47–60% in the 340M models and 69% in GDN-1.3B; edit strength falls by roughly half or more. GLA editing points were not evaluated at some conditions.Source: §4.2 · p. 8, Figure 8 · p. 8
  • The authors suggest GDN’s delta-rule correction may suppress some accumulated cross-talk, whereas GLA’s gated additions may leave overlapping contributions superposed.They characterize update-rule effects as a possible explanation, not a definitive mechanism.Source: §4.2 · p. 8

5 RELATED WORK

Related work frames recurrent linear attention as a distinct memory substrate from token-addressable KV caches. This paper builds on causal analysis and retrieval studies by examining how pretrained recurrent states store and retrieve facts under competition.

Linear attention and recurrent memory

  • Prior recurrent-memory work studies fast-weight or associative-memory interpretations and develops gating and delta-rule architectures; this paper instead causally examines how pretrained models write, retain, and retrieve individual facts (Schlag et al., 2021; Yang et al., 2024a;b; 2025; Arora et al., 2024).The cited work also studies retrieval capability, capacity, and architectural design.Source: §5 · pp. 8–9

Memory efficiency in attention models

  • KV-cache methods exploit token-addressable entries for eviction, compression, or quantization, whereas recurrent state combines past inputs into a fixed-size representation that cannot be inspected or removed token by token.The paper studies selective access, interference, and the return to token-addressable memory in hybrid models.Source: §5 · p. 9

Causal analysis of model memory

  • Prior causal tracing and activation-patching work examines factual recall and in-context learning; this study intervenes on recurrent writes, states, and query-time outputs to separate memory writing from later access.The interventions also test how multiple facts interact within shared state.Source: §5 · p. 9

6 DISCUSSION AND CONCLUSION

The discussion characterizes recurrent state as structured shared memory: facts can remain selectively accessible, but their causal support is coupled and interference grows with competition. The authors delimit this conclusion to controlled associative recall and a limited set of checkpoints, and point to application-driven memory management as future work.

6 DISCUSSION AND CONCLUSION

  • Recurrent state is better viewed as structured shared memory than compressed token-specific KV entries: facts remain selectively accessible, but their causal support is coupled, helping explain interference under competition.When full-attention KV memory is available in tested hybrids, that state predominantly carries recalled identity.Source: §6 · p. 9
  • The experiments cover controlled associative recall, not all long-context reasoning or generation, and use a limited set of pretrained GLA and Gated DeltaNet checkpoints.The pure–hybrid comparison uses separately trained released models; concentrated pathways should not be treated as isolated memory locations.Source: §6 · p. 9
  • Future work proposes testing memory management in application-driven settings, including selectively gating, refreshing, or partitioning recurrent writes and choosing what remains in KV memory.The authors motivate these directions by the need to control competition among accumulated memories.Source: §6 · p. 9
  • GPT assisted with language editing and Codex with software implementation; the authors report reviewing the outputs and did not use generative AI for other research tasks.They take responsibility for the final content and artifacts.Source: AI USE STATEMENT · p. 10
  • The authors state that datasets and pretrained checkpoints are public, with experimental protocols in the main text and additional implementation details in the appendix.These materials specify the conditions and procedures used for the reported analyses.Source: REPRODUCIBILITY STATEMENT · p. 10

A MODEL CHECKPOINTS AND EXPERIMENTAL SETUP

The appendix identifies pretrained checkpoints, model configurations, and inference settings used in the experiments. The pure–hybrid GDN pair is near-matched rather than a controlled architectural retraining.

A MODEL CHECKPOINTS AND EXPERIMENTAL SETUP

  • Experiments evaluate base pretrained variants without parameter updates: GLA-340M, GDN-340M, GDN-1.3B, hybrid GDN-340M, and Qwen3.5-4B-Base.The cited checkpoint entries include commit hashes.Source: A MODEL CHECKPOINTS AND EXPERIMENTAL… · pp. 12–13
  • Table 1 summarizes architectural configurations; maximum context follows released model configurations, while experiment protocols set sequence length.For Qwen3.5, key and value head counts are reported separately.Source: Table 1 · p. 13
  • Pure GLA and GDN checkpoints have recurrent blocks in all 24 layers; GDN uses a width-four causal short convolution, unlike GLA.The stated recurrent state is 256 × 256 per head in the GDN checkpoints.Source: A MODEL CHECKPOINTS AND EXPERIMENTAL… · p. 13
  • The M-A-P hybrid replaces eight of 24 recurrent blocks with full-attention blocks; it shares stated depth, width, tokenizer, and training budget with GDN-340M, but is not a controlled retraining comparison.The released pair differs in parameter count by approximately 4.2%, and metadata do not establish identical data order, optimizer state, or random seed.Source: A MODEL CHECKPOINTS AND EXPERIMENTAL… · p. 13
  • Qwen3.5-4B and 9B use 32-layer repeating three-GDN/one-full-attention layouts; experiments use their text-only language-model backbones.The active text models contain approximately 4.206B and 8.954B parameters, respectively.Source: A MODEL CHECKPOINTS AND EXPERIMENTAL… · p. 13
  • Inference uses disabled gradients, BF16 weights and standard activations, and FP32 recurrent caches; Qwen3.5-9B runs across two GPUs without CPU offloading.The passage specifies the software stack and two GeForce RTX 4080 SUPER GPUs.Source: A MODEL CHECKPOINTS AND EXPERIMENTAL… · pp. 13–14

B SELECTION OF RECURRENT HEADS FOR CAUSAL ANALYSIS

The paper selects recurrent heads by measuring how donor-state substitutions shift answer margins, then checks the selection’s stability and fixes the selected heads before confirmation analyses. It also frames held-out tests to assess whether these pathways remain important beyond the development set.

B SELECTION OF RECURRENT HEADS FOR CAUSAL ANALYSIS

  • Replacing a head’s pre-query state with its donor state ranks it by the mean shift in donor-versus-recipient logit margin across 16 development sources.The highest-scoring head is unique for each model, and donor-answer accuracy does not affect the ranking.Source: B SELECTION OF RECURRENT HEADS FOR… · p. 14
  • The selected head has the largest intervention effect on 16/16 sources for GLA-340M and GDN-340M, and 15/16 for GDN-1.3B.Leave-one-source-out recomputation preserves the same mean-score winner in all three models; the 16 groups are token windows, not necessarily distinct articles.Source: B SELECTION OF RECURRENT HEADS FOR… · p. 14
  • The 340M development screen independently recovers the two previously selected heads, while the 1.3B screen prospectively selects its head before confirmation.Selected heads remain fixed for subsequent write, read, and multi-fact interventions.Source: B SELECTION OF RECURRENT HEADS FOR… · p. 14

C ADDITIONAL VALIDATION OF RECURRENT MEMORY PATHWAYS

  • Held-out validation maps all recurrent components at writing and reading, then compares those causal localizations with an independent depth-wise analysis of answer-related readout.The two tests probe whether selected components remain exceptional outside development and where answer-related information becomes accessible across depth.Source: C ADDITIONAL VALIDATION OF RECURRENT… · p. 15

C.1 HELD-OUT CAUSAL LANDSCAPE

On held-out sources, the selected recurrent components remain exceptional at both fact-time writing and pre-query reading, although importance across other components is weakly aligned across those stages. A separate layerwise analysis locates sharp increases in accessible answer readout at the same hub layers.

C.1 HELD-OUT CAUSAL LANDSCAPE

  • GLA L21H3 and GDN L20H2 rank first among all 96 components at both writing and reading, with rank intervals [1, 1] in all four cases.Only 1.0% of GLA components and 3.1% of GDN components exceed the 0.25-logit threshold at both stages.Source: §C.1 · p. 15, Figure 10 · p. 16
  • Write–read rank correlations are 0.173 [−0.014, 0.320] for GLA and 0.149 [−0.027, 0.285] for GDN, indicating weak alignment beyond the selected components.The analysis maps all 96 recurrent components on 32 held-out sources using distinct fact-time and pre-query interventions.Source: §C.1 · pp. 15–16

C.2 LAYERWISE LOCALIZATION OF ANSWER READOUT

  • On confirmation sources, retention readout rises from 28.9% to 95.3% at GLA layer 21 and from 25.8% to 100% at GDN layer 20 across the recurrent computation.After three updates, the corresponding increases are 28.9 and 43.0 percentage points.Source: §C.2 · p. 16
  • The layerwise transition localizes accessible readout, not necessarily where information is first formed, because a logit lens may miss other bases, tokens, or internal states.The readout transitions occur at layers containing components independently identified by state interventions.Source: §C.2 · p. 16

D.1 SELECTIVE SUPPRESSION OF A SINGLE ASSOCIATION

A neutral-state intervention strongly suppresses the queried association while largely preserving accuracy on other facts, consistent with selective access despite cross-fact coupling. The selected head also functions as a reusable pathway across task variants rather than as a fixed semantic slot.

D.1 SELECTIVE SUPPRESSION OF A SINGLE ASSOCIATION

  • Replacing the selected head’s pre-query state with a neutral-state counterpart lowers target accuracy from 91.8% to 7.6% in GLA and from 96.1% to 37.1% in GDN.Mean accuracy on the other three facts changes by only +2.15 and +1.11 percentage points, respectively.Source: §D.1 · pp. 16–17, Figure 12 · p. 17
  • Non-target margins shift more than under norm-matched random controls, so preserved non-target accuracy does not mean facts are fully independent.A small number of previously correct GLA answers are disrupted even as mean non-target accuracy increases.Source: §D.1 · p. 17
  • The neutral-state intervention is a diagnostic, not a practical deletion procedure, because it uses oracle knowledge and combines a neutral head state with the clean execution’s remaining state.The authors make no claim of practical unlearning, privacy protection, or guaranteed memory removal.Source: §D.1 · p. 17

D.2 A REUSABLE PATHWAY RATHER THAN A FIXED SEMANTIC SLOT

  • The same head remains causally effective across binding counts, templates, value vocabularies, and updated-value retrieval, but effect strength varies by model and condition.GLA donor-answer rates remain at least 89.7%; GDN ranges from 46.8% in the update condition to 91.9% under the colon template.Source: §D.2 · p. 18, Figure 13 · p. 18
  • A rank-4 basis learned from color examples transfers the color donor effect but not shape answers, whose effect mostly lies outside that basis.This small post-confirmation diagnostic within the four-choice task family does not establish broader generalization of task-dependent directions.Source: §D.2 · p. 18, Figure 13 · p. 18

E.1 ROBUSTNESS OF HYBRID KV DOMINANCE

In the tested hybrid settings, KV-only replacement recovers nearly all of the donor-margin effect, while recurrent-only replacement produces no donor answers. Additional pure recurrent checkpoints show markedly different read-path concentration, and the comparisons do not establish a scaling law.

E.1 ROBUSTNESS OF HYBRID KV DOMINANCE

  • Across three additional hybrid settings, recurrent-only replacement produces no donor answers, whereas KV-only replacement recovers 96.2–97.2% of the full donor-margin effect.The settings include natural entity updates with WikiText or PG19 filler and RULER single-NIAH at approximately 4K tokens.Source: §E.1 · pp. 18–19, Table 2 · p. 19

E.2 READ-PATH CONCENTRATION ACROSS ADDITIONAL CHECKPOINTS

  • Read-path concentration differs substantially across pure recurrent checkpoints: GLA-7B reaches 92.3% recovery with its top 16 components, while RWKV7-2.9B reaches 97.8% with its top 32.RWKV7-13.3B has 53.1% recovery with its leading component and reaches 91.6% by the top 32.Source: §E.2 · p. 19
  • The checkpoint profiles are not monotonic with parameter count, and architecture, training procedure, model scale, and task format vary jointly, precluding a simple scaling interpretation.The additional-checkpoint analysis probes pre-query read-path concentration, not the full write, persistent-state, and read lifecycle.Source: §E.2 · p. 19

F ADDITIONAL DIAGNOSTICS AND SCOPE

  • The final diagnostics ask whether useful state information can remain available when the native model answers incorrectly and describe a prose-retrieval setting with insufficient baseline competence for the planned causal analysis.The supplied passage states the diagnostic questions and scope, not their outcomes.Source: F ADDITIONAL DIAGNOSTICS AND SCOPE · p. 20

F.1 STATE FEATURES RECOVER A SUBSET OF NATIVE ERRORS

Source-disjoint correction models recover some GLA native errors using recurrent-state features, though correction effects vary across architectures. In the prose-retrieval pilot, baseline competence was insufficient for causal interventions, leaving extension of the memory lifecycle to prose retrieval open.

F.1 STATE FEATURES RECOVER A SUBSET OF NATIVE ERRORS

  • After changing the value vocabulary, template, and source range, GLA accuracy increases from 30.5% to 49.2%, with 53 recoveries and 29 disruptions.The figure labels also report recovered native errors minus disrupted originally correct cases.Source: §F.1 · p. 20, Figure 15 · p. 20
  • GLA accuracy increases from 24.2% to 52.3%, recovering 64 native errors while disrupting 28 originally correct cases.Figure 15 reports corrected accuracy with 95% confidence intervals.Source: Figure 15 · p. 20, §F.1 · p. 20
  • Correction effects vary by architecture: GDN changes from 46.9% to 48.4%, while the tested Qwen recurrent feature reduces accuracy from 78.9% to 46.9%.The authors limit the GLA interpretation to some failures occurring despite answer-related information remaining accessible in recurrent-state features.Source: §F.1 · p. 20
  • Some GLA errors occur despite accessible answer-related state features; this does not show every error is a readout failure or that the feature is the stored answer.The four-choice correction procedure is not established as a general decoding method.Source: §F.1 · pp. 20–21

F.2 COMPETENCE CHECK FOR PROSE RETRIEVAL

  • The prose pilot required at least 80% full-vocabulary top-1 accuracy across two query paraphrases before interventions; it tested 512 filler tokens, with a prespecified 128-token fallback.Table 3 reports joint top-1 accuracy across both paraphrases, clustered over the same 32 source families.Source: §F.2 · p. 21, Table 3 · p. 21
  • Neither prose configuration met the competence threshold, so donor-write, state-swap, and mediation interventions were not applied.This is an unmet prerequisite, not a negative result about the memory lifecycle; extension to prose retrieval remains open where task competence is sufficient.Source: Table 3 · p. 21, §F.2 · p. 21

Related papers

Loading 2609.33093v1…