RunningTensor: Generalizing Linear Attention to Higher-Order Recurrent States

• Linear attention and state-space models replace quadratic sequence-length costs with linear recurrences, but their matrix state limits memory and expressivity.Source: §1 · p. 1
• RunningTensor uses an order-o tensor state, updated by a rank-1 outer product and read through o−1 vector queries; order 2 recovers linear attention.Source: §1 · pp. 1–2
• State size scales from O(W) for vector RNNs to O(W²) for linear attention and O(W^o) for order-o RunningTensor, while a full pass remains linear in T.Source: §1 · p. 2
• The instantiated model adds data-dependent forget gates, query-skip gates, and ConcOnlyC, a near-parameter-free channel mix.Source: §1 · p. 2
• RunningTensor improves over linear-attention and SSM baselines on synthetic multi-query recall and improves on several language-understanding and non-synthetic retrieval downstreams after pretraining.Source: §1 · p. 2

2.1 Preliminary: Linear Attention

• The linear-attention formulation has a recurrent view for linear-time inference and a matrix view for parallel training, with causality enforced by a lower-triangular mask.The recurrent and unrolled forms are equivalent.Source: §2.1 · p. 2

2.2 The RunningTensor architecture

• The RunningTensor uses a third-order state read by contracting two modes with vector queries; order 2 reduces to causal linear attention, and the construction extends to higher orders.The paper restricts its implementation to order 3 for simplicity.Source: §2.2 · pp. 2–3
• Data-dependent forgetting reweights past context, while query-skip gates let the readout use both state axes, one axis, or neither.Expanded queries and keys are L2-normalized to keep pairwise dot products bounded and improve delta-rule stability.Source: §2.2 · p. 3
• The tensor delta rule subtracts the prior state's prediction from the value before applying an overwrite gate, and uses chunk-parallel training with stepwise recurrent inference.The subtracted prediction removes the component already represented along the expanded key and query pair.Source: §2.2 · p. 3

2.3 Training and Optimization

• All models use next-timestep prediction and cross-entropy under matched 0.6B-parameter training conditions on FineWeb-Edu 100BT [18].The learning rate was selected using one day of Qwen training and then shared across models.Source: §2.3 · p. 4

3.1 Synthetic retrieval (MQAR)

• RunningTensor matches perfect MQAR F1 across evaluated lengths with fewer parameters than attention, while remaining faster than attention and the best SSM.Matrix-state and second-order baselines degrade as context grows; ∆Net falls short of the tensor delta variant at 2048 tokens.Source: §3.1 · p. 4, Table 1 · p. 5

3.2 Component ablation

• Table 2 compares RunningTensor variants on MQAR validation F1 at dmodel = 128 under the Table 1 protocol.The text distinguishes GDN from the full architecture by whether gout passes through the temporal filter.Source: Table 2 · p. 5, §3.2 · p. 5

Linear Attention and Fast Weight Mechanisms

• Linear attention uses kernel-based similarity for associative computation, while Performer approximates softmax attention with random features [4] [20] [21].RetNet adds decay and rotational structure, and Gated Linear Attention incorporates learnable gating [22] [23].Source: §4 · p. 5

State Space Models and Gated Linear RNNs

• State-space models evolved from fixed or structured transitions toward input-dependent gating, while related linear recurrent architectures share similar computational principles [5] [31] [32] [33] [34] [36].Selective state updates remove dependence on previous hidden states to preserve parallelism [38] [39].Source: §4 · pp. 5–6

Delta Rule, Online Learning, and Hybrid Architectures

• The delta rule offers greater memory capacity than Hebbian-style updates, while stability and scalability challenges motivate parallelization, normalization, and structured approximations [12] [13] [41].Hybrid architectures combining attention with linear recurrent or SSM layers are presented as a practical balance between efficiency and performance [45].Source: §4 · p. 6