Hierarchically Gated Recurrent Neural Network for Sequence Modeling

1 Introduction

• RNNs offer linear complexity but suffer from slow sequential training and limited long-term dependency modeling, whereas Transformers improve both capabilities at quadratic computational cost.These trade-offs motivate revisiting efficient RNN variants for long-sequence modeling.Source: §1 · p. 1
• Element-wise linear recurrence removes recurrent nonlinearities for parallelized training and replaces matrix multiplication with efficient element-wise updates.This approach underlies modern linear RNNs and related state-space models.Source: §1 · p. 1
• Existing linear RNNs often gate recurrence outputs while overlooking forget gates, despite forget gates being central to retaining or discarding information.HGRN therefore adopts gated linear recurrence with explicit forget gates.Source: §1 · p. 2
• HGRU adds a learnable lower bound to forget gates and increases that bound monotonically across layers, assigning shorter-term modeling to lower layers and longer-term modeling to upper layers.The lower bound also pushes gates away from saturated regimes, while the paper reports strong results across language modeling, image classification, and long-range benchmarks.Source: §1 · p. 2

Efficient token mixing for sequence modeling

• Long convolutions and linear recurrence offer better asymptotic efficiency than self-attention, but long convolutions require historical caching and retain higher inference complexity than RNNs.These limitations make linear RNNs a suitable alternative for long-sequence token mixing.Source: §2 · p. 2

Element-wise linear recurrence

• Element-wise linear recurrence accelerates hidden-state updates and enables parallel scan training by removing recurrent nonlinearities, with stacked MLP blocks restoring nonlinear modeling capacity.Prior work also supports the universal-approximation potential of this architecture.Source: §2 · p. 2

Linear recurrence

• Linear recurrence separates exponential moving averages with static decay from gating schemes with data-dependent decay, whereas HGRN uses dynamic decay through forget gates.HGRN supports parallel scans but does not use the convolutional view required for FFT-based training.Source: §2 · p. 3

Linear Transformers and linear RNNs

• Linear Transformers can be viewed as RNNs with forget values fixed to one, which limits their ability to discard irrelevant information and causes attention dilution.Prior remedies include delta-rule forgetting and gating mechanisms.Source: §2 · p. 3

Long-term dependencies in RNNs

• HGRN combines gating with regularized dynamic decay rates approaching one in upper layers to address long-term dependency and gradient-vanishing issues.This extends common approaches based on gating, eigenvalue initialization, or skip connections.Source: §2 · p. 3

3.1 Architecture overview

• HGRN stacks HGRU token mixers and GLU channel mixers in multiple layers.Each HGRU performs linear recurrence, while GLU provides channel mixing.Source: §3.1 · p. 3, Figure 1 · p. 4

3.2 HGRU exploration

• HGRU uses input-dependent forget gates whose values depend only on the current input, enabling parallel-scan computation.The recurrence uses element-wise products rather than full matrix multiplication.Source: §3.2 · p. 3
• Complex-valued recurrence uses λ_t for historical-information intensity and θ for oscillation frequency, with θ shared across time steps.The data-independent phase parameterization supports interpreting relative position information.Source: §3.2 · p. 4
• Layerwise cummax creates monotonically increasing forget-gate lower bounds so upper layers can model long-range dependencies while retaining the ability to forget irrelevant information.The lower bound is applied across layers rather than hidden-state dimensions as in ON-LSTM.Source: §3.2 · p. 4
• The final HGRU ties input and forget gates, adds an output gate before projection, and achieves the best performance among the tested gate configurations.Removing either the input or output gate degrades the reported comparison.Source: §3.2 · p. 5

3.3 Token mixing perspective of HGRU

• The token-mixing matrix decomposes into Λ for long-distance dependencies and a Toeplitz Θ for relative positional relationships and enhanced expressiveness.This gives HGRU a RoPE-enhanced attention interpretation in which Λ uses cumulative data-dependent decay rates.Source: §3.3 · p. 5

4.2 Results

• HGRN performs better at 1b scale and matches Transformer-based models on downstream tasks while consuming only 1/3 of the tokens.Source: §4.2 · p. 7
• HGRN achieves the best performances on the Long Range Arena benchmark and outperforms all competing methods.Source: §4.2 · p. 7, Table 6 · p. 8
• With comparable parameter sizes, HGRN outperforms TNN and vanilla Transformer on ImageNet-1K image classification.Source: §4.2 · p. 8, Table 7 · p. 8

4.3 Ablation Study

• Removing forget gates significantly reduces HGRN performance, while adding data-dependent forget gates improves LRU and outperforms data-independent gates.Source: Table 8 · p. 8, §4.3 · p. 9
• The complete input-gate, output-gate, and tied input-forget-gate design achieves the best autoregressive language-modeling performance.Source: §4.3 · p. 9, Table 9 · p. 9
• Monotonically increasing lower bounds provide the largest improvement among lower-bound variants, assigning broader dependency modeling to upper layers.Source: §4.3 · p. 9, Table 10 · p. 9
• Lower bounds are essential for long-sequence tasks, where models without them perform poorly or sometimes fail to converge, whereas language modeling remains strong without them.Source: §4.3 · p. 9, Table 13 · p. 10

4.4 Analysis on forget gate values

• Complex-valued recurrence improves performance, while making its phase argument data-dependent prevents the recurrence matrix from capturing relative information.Source: §4.3 · pp. 9–10, Table 11 · p. 10

• Gated linear RNNs achieve impressive performance across different tasks and modalities without compromising efficiency, while forget gates improve linear RNN modeling.The paper emphasizes an additive lower bound on forget gate values for modeling long-term dependencies.Source: §5 · p. 10

• HGRN’s empirical evaluation remains smaller in scale than evaluations of other large-scale models.Source: Limitations and broader impact · p. 10

6.1 Extrapolation test

• HGRN can train on short sequences and test on longer sequences in the extrapolation evaluation.Table 14 reports extrapolation performance for competing methods.Source: §6.1 · p. 16, Table 14 · p. 19

6.2 Speed comparison

• HGRN achieves state-of-the-art training and inference speed on the Long Range Arena benchmark.Table 15 compares speed at input lengths from 1K through 5K and marks out-of-memory methods with dashes.Source: §6.2 · p. 16, Table 15 · p. 20

6.3 Visualization

• The appendix visualizes forget rates for each layer of a language-model-trained HGRN.Source: §6.3 · p. 16, Figure 3 · p. 17

6.4 Configurations

• The appendix lists detailed training configurations for the reported experiments.Table 16 defines total batch size and abbreviations for autoregressive language modeling and image modeling.Source: §6.4 · p. 16, Table 16 · p. 20
• The appendix visualizes the token-mixing matrix in each layer.Source: Figure 4 · p. 18