Source-linked AI summary
GateLoop: Fully Data-Controlled Linear Recurrence for Sequence Modeling
Tobias Katsch
TL;DR
Existing linear recurrent models do not fully exploit data-controlled state transitions, motivating GateLoop as a generalization of linear recurrence. GateLoop combines efficient recurrent and parallel computation with an attention-equivalent formulation, and it outperforms established models for autoregressive language modeling.
Problem
Existing linear recurrent models use only a special case of linear recurrence, limiting the use of data-controlled state transitions for sequence modeling.
Source: §1 · pp. 1–2Method
Results
GateLoop outperforms Transformer, Hyena, and S5-Hyena on WikiText103 autoregressive language modeling.
Source: §1 · p. 2Takeaways & Limitations
GateLoop combines O(l) recurrent execution, O(l log l) parallel training, and an O(l^2) attention-equivalent mode with data-controlled relative-positional information.
Source: §1 · p. 2, §7 · p. 9The paper leaves initialization strategies, amplitude and phase activations, and interpretability of learned state transitions for future research.
Source: §6 · p. 9
Abstract
from arXiv · showhide
Linear Recurrence has proven to be a powerful tool for modeling long sequences efficiently. In this work, we show that existing models fail to take full advantage of its potential. Motivated by this finding, we develop GateLoop, a foundational sequence model that generalizes linear recurrent models such as S4, S5, LRU and RetNet, by employing data-controlled state transitions. Utilizing this theoretical advance, GateLoop empirically outperforms existing models for auto-regressive language modeling. Our method comes with a low-cost $O(l)$ recurrent mode and an efficient $O(l \log_{2} l)$ parallel mode making use of highly optimized associative scan implementations. Furthermore, we derive an $O(l^2)$ surrogate attention mode, revealing remarkable implications for Transformer and recently proposed architectures. Specifically, we prove that our approach can be interpreted as providing data-controlled relative-positional information to Attention. While many existing models solely rely on data-controlled cumulative sums for context aggregation, our findings suggest that incorporating data-controlled complex cumulative products may be a crucial step towards more powerful sequence models.
1 INTRODUCTION
GateLoop addresses limitations of recurrent and attention-based sequence models by generalizing linear recurrence with data-controlled gating, while retaining efficient computation. It also connects this formulation to attention and outperforms several established models on WikiText103.
Motivation
- Transformers offer parallel training and global pairwise dependencies but incur O(l^2) complexity, motivating efficient attention replacements.Reduced-complexity alternatives such as Reformer, Linformer, and Performer have not matched the original Transformer’s practical performance in the cited discussion.Source: §1 · p. 2
Related work
- Linear state-space models such as S4, S4D, S5, and LRU improve long-sequence efficiency but involve specialized or carefully designed initialization schemes.S4-family models exploit linear time-invariant structure, while LRU simplifies initialization relative to S4, S4D, and S5.Source: §1 · p. 2
Contributions
- GateLoop generalizes existing linear recurrent models by using data-controlled gating of inputs, hidden states, and outputs, with O(l log l) parallel training.This extends the restricted linear-recurrence cases used by existing models and uses associative scans for efficient computation.Source: §1 · p. 2
- GateLoop derives an equivalent O(l^2) formulation that provides data-controlled relative-positional information to attention.The formulation links GateLoop to Transformer while preserving the interpretation of data-controlled state transitions.Source: §1 · p. 2
- GateLoop outperforms Transformer, Hyena, and S5-Hyena on WikiText103 autoregressive language modeling.This is the paper’s reported empirical comparison against these state-of-the-art models.Source: §1 · p. 2
2 PRELIMINARIES
The preliminaries define causal sequence-to-sequence modeling and linear recurrent layers, then describe state-space models, their diagonalization and discretization, and initialization choices used by representative systems.
2 PRELIMINARIES
- Causal sequence-to-sequence models produce each y_n using only inputs x_m with m ≤ n, enabling autoregressive language modeling.The input and output sequences are represented as multichannel matrices X and Y.Source: §2 · p. 2
2.1 RECURRENT NEURAL NETWORK
- An RNN becomes linear when its activation function is the identity instead of tanh or sigmoid.The general recurrent layer uses learnable transition, input, and output parameters together with an activation function.Source: §2.1 · p. 3
2.2 STATE SPACE MODEL
- SSMs use complex-valued state-space parameters, and diagonalizing the transition matrix converts the system into an equivalent diagonal representation.The eigenvalue decomposition absorbs the eigenvector matrices into the input and output parameters.Source: §2.2 · p. 3
- Zero-order-hold discretization converts the continuous SSM into a linear recurrence with A = exp(∆Λ), B = Λ^-1(A − I)B̄, and C = C̄.The discretization step size controls how rapidly the state changes and therefore affects retain/forget behavior.Source: §2.2 · p. 3
- SSM families use distinct transition initializations: HiPPO-based methods support S4, DSS, S4D, and S5, while LRU and RetNet use alternative formulations.The preliminaries also specify S4D, LRU, and RetNet transition parameterizations and note that the initial hidden state is zero.Source: §2.2 · pp. 3–4
3 DATA CONTROLLED LINEAR RECURRENCE
GateLoop extends linear recurrence with data-controlled input, output, and state-transition gates, connecting efficient recurrent computation to surrogate attention. It generalizes several existing models while offering associative-scan and attention-style computation modes.
3 DATA CONTROLLED LINEAR RECURRENCE
- GateLoop introduces content-aware state transitions that control forgetting and retention alongside input and output gating.The state transition a_n is diagonal and time-varying, unlike static transitions in existing linear recurrent models.Source: Figure 2 · p. 4, §3 · p. 4
3.1 RELATION TO OTHER MODELS
- S4, S4D, and LRU arise as special cases without content-aware gating or data-controlled state transitions, while RetNet fixes the transition gate.Hyena and GateLoop are described as mutually exclusive generalizations of the linear RNN layer.Source: §3.1 · p. 5
3.2 EFFICIENT ASSOCIATIVE SCAN COMPUTATION
- GateLoop computes its non-convolutional recurrence in O(l log2 l) parallel time using an associative scan over transition and input-state tuples.The scan applies an associative binary operator to produce parallel prefix results, though its working memory can bottleneck for large sequences.Source: §3.2 · p. 5
3.3 SURROGATE ATTENTION REPRESENTATION
- An equivalent O(l2) formulation rewrites cumulative state-transition products into prefix products and applies a causal mask to obtain surrogate attention.This formulation exposes GateLoop’s data-controlled relative-positional information interpretation, but cumulative products risk underflow.Source: §3.3 · p. 6, Figure 3 · p. 6
3.4 GENERALIZING SOFTMAX-ATTENTION
- For softmax attention, masking upper-triangular relative-positional scores with −∞ and applying softmax yields the desired data-controlled reweighting.The O(l2) representation also supports generalization to other nonlinear attention forms.Source: §3.4 · p. 6
4 PRACTICAL IMPLEMENTATION
The practical GateLoop model uses parallel scans, projected inputs, and separate magnitude and phase control for state transitions. Experiments use sigmoid magnitude activation rather than LRU’s stable-exponential activation.
4 PRACTICAL IMPLEMENTATION
- Experiments use the parallel-scan computation, with linear projections producing values, keys, and queries from each input.Magnitude and phase of state transitions are controlled separately, following prior approaches.Source: §4 · p. 7
- LRU parameterizes transition magnitude as exp(−exp(α)), restricting |a| to (0, 1) and preventing long-horizon blow-up.This parameterization is non-data-controlled and is presented as the baseline activation design motivating the practical choice.Source: §4 · p. 7
- Sigmoid is chosen for transition magnitudes because LRU’s stable-exponential activation is biased toward amplitudes close to 1.The phase activation is omitted because the imaginary part of an individual transition need not be restricted to a specific interval.Source: Figure 4 · p. 7, §4 · p. 7
5 EXPERIMENTAL RESULTS
Experiments test whether data-controlled state transitions improve sequence modeling on a synthetic memory task and WikiText103. GateLoop learns input-dependent forgetting, maintains accuracy over longer memory spans, and achieves a significant performance advance over existing models.
5 EXPERIMENTAL RESULTS
- The experiments evaluate data-controlled state transitions using an interpretable synthetic language task and autoregressive modeling on WikiText103.The synthetic task is designed to expose benefits of input-dependent forgetting, while WikiText103 provides a natural-language evaluation.Source: §5 · p. 7, §5.1 · p. 7, §5.2 · p. 8
5.1 MEMORY HORIZON
- Near reset positions, GateLoop applies close-to-zero state transitions, effectively vacating its hidden state for new relevant information.Figure 5 visualizes transition magnitudes across layers, channels, and sequence positions.Source: Figure 5 · p. 8
- Input-dependent forgetting and retention significantly improve test accuracy compared with an otherwise similar model using a fixed state transition.Figure 6 isolates the effect of replacing the fixed transition with an input-dependent mechanism.Source: Figure 6 · p. 8
- Across all required memory spans, the fully data-controlled variant outperforms the fixed variant and remains comparable for twice as long before performance degrades.The fixed variant falls rapidly after the required memory span exceeds 50, whereas the fully data-controlled variant maintains comparable performance for twice that span.Source: Figure 7 · p. 8
5.2 WIKITEXT103
- WikiText103 contains over 100 million tokens from verified Wikipedia articles and is used to test the fully data-controlled model against state-of-the-art competitors.The model details are reported separately in section C.Source: §5.2 · p. 8
- GateLoop achieves a significant performance leap over existing models on WikiText103 while avoiding softmax attention, tedious initialization, and long implicit convolutions.The comparison uses WikiText103 test perplexity with a shared tokenizer; results for other models come from prior work.Source: §5.2 · p. 9, Table 1 · p. 9
6 FUTURE WORK
Future work focuses on extending the groundwork for fully data-controlled linear RNNs. Proposed directions include initialization, activation choices, and interpretability of learned state transitions.
6 FUTURE WORK
- Future research should examine initialization strategies, amplitude and phase activations, and the interpretability of learned state transitions.The authors identify these as opportunities beyond the paper’s foundational focus.Source: §6 · p. 9
7 CONCLUSION
The paper introduces GateLoop as a fully data-controlled linear RNN that generalizes existing linear recurrent models. It combines efficient recurrent and parallel computation with an equivalent attention interpretation and strong autoregressive language-modeling performance.
7 CONCLUSION
- GateLoop generalizes existing linear recurrent models through data-controlled gating of inputs, outputs, and state transitions.The model is presented as a fully data-controlled linear RNN.Source: §7 · p. 9
- GateLoop has linear recurrent complexity O(l) and an efficient parallelizable training strategy with O(l log l) complexity using parallel scans.The conclusion distinguishes recurrent runtime from the parallel training strategy.Source: §7 · p. 9
- An equivalent O(l^2) surrogate-attention formulation interprets GateLoop as providing relative positional information to attention.This formulation connects the recurrent mechanism to attention without changing the stated equivalence.Source: §7 · p. 9
- The paper validates fully data-controlled linear recurrence as highly performant for autoregressive language modeling.The conclusion states the empirical validation at the paper level without restricting it to a single benchmark.Source: §7 · p. 9
A MEMORY HORIZON DATASET DETAILS
The Memory Horizon Dataset is designed to test whether linear recurrent models can control forgetting based on reset tokens. Its targets compress the post-reset input history through an alternating sum of products modulo a specified number.
A MEMORY HORIZON DATASET DETAILS
- The Memory Horizon Dataset evaluates whether data-controlled state transitions outperform non-data-controlled transitions by requiring memory of inputs since the latest reset token.Each sample is specified by an input number range, reset token, sequence length, and number of randomized resets.Source: A MEMORY HORIZON DATASET DETAILS · p. 11
- The dataset vocabulary contains a reset token and number tokens for inputs, while outputs contain number tokens from 0 through the maximal output number.These construction parameters are listed in Table 2.Source: Table 2 · p. 11
- Targets are computed from the sequence since the most recent reset by alternating additions and subtractions of products formed from opposite-end input pairs.If the list has odd length, the middle element is added or subtracted according to the current operation.Source: A MEMORY HORIZON DATASET DETAILS · p. 11
- The compressed memory value is obtained by taking the resulting alternating sum of products modulo a specified number.This operation produces the target token for the autoregressive task.Source: A MEMORY HORIZON DATASET DETAILS · p. 11
B PARALLEL SCAN
GateLoop’s parallel scan uses an associative binary operator, enabling efficient parallel computation of the recurrence.
B PARALLEL SCAN
- The GateLoop operator is computed through a parallel scan over the first four sequence elements.Source: Figure 9 · p. 12
- Associativity follows because both evaluation orders produce the same combined state-transition and accumulated-output terms.Source: B PARALLEL SCAN · p. 12
C MODEL DETAILS
Each GateLoop model layer combines temporal aggregation with channel-wise function approximation, while skip connections and normalization support stable information flow.
C MODEL DETAILS
- Each layer uses a GateLoop Time-Mixing block to aggregate information across time with content-aware inputs and real-valued projections.Source: C MODEL DETAILS · p. 12
- A point-wise feed-forward network provides Channel-Mixing to approximate functions along the channel dimension.Source: C MODEL DETAILS · p. 12
- Skip connections and Layer Normalization let information bypass mixing blocks and stabilize training.Source: C MODEL DETAILS · p. 12
- The model stacks L layers between a learned input token embedding and a language head that maps final outputs to vocabulary logits.Source: C MODEL DETAILS · p. 12
- The full model architecture is visualized in Figure 10.Source: Figure 10 · p. 13
C.1 MEMORYHORIZON HYPERPARAMETERS
The appendix lists separate hyperparameter configurations for the MemoryHorizon and WikiText103 experiments.
C.1 MEMORYHORIZON HYPERPARAMETERS
- Table 3 reports the model hyperparameters used for the MemoryHorizon experiment.Source: Table 3 · p. 13
C.2 WIKITEXT103 HYPERPARAMETERS
- Table 4 reports WikiText103 hyperparameters, including a smaller learning rate and no weight decay for state-transition projections.Source: Table 4 · p. 14