Were RNNs All We Needed?

• RNNs such as LSTMs and GRUs were widely used for sequence modelling, but their sequential nature limited parallelization and slowed training on long sequences.Source: §1 · p. 1
• Transformers introduced parallelizable self-attention and broad domain success, but their quadratic complexity makes long-sequence modelling expensive.Source: §1 · pp. 1–2
• Recent alternatives include state-space models, linearized attention, and linear recurrent models that parallelize training and scale more efficiently.Source: §1 · p. 2
• Removing previous-state dependencies from LSTM and GRU gates yields minLSTMs and minGRUs with fewer parameters, fully parallelizable training, and competitive performance.The paper presents these simplified models as lightweight alternatives to increasing architectural complexity.Source: §1 · p. 2

2 Background

• Vanilla RNNs capture temporal dependencies but can suffer vanishing and exploding gradients, limiting long-term dependency learning.Source: §2 · p. 2

2.1 LSTM

• LSTMs use input, forget, and output gates plus separate cell and hidden states to control information flow and support long-term dependencies.The cell state carries information across time, while the gates regulate additions, deletions, and emitted hidden-state information.Source: §2.1 · pp. 2–3

2.2 GRU

• GRUs reduce LSTM’s complexity to two gates and one hidden state, requiring O(3dh(dx + dh)) parameters while retaining competitive performance.GRUs combine LSTM’s input and forget gates into an update gate and use a reset gate for candidate-state computation.Source: §2.2 · p. 3
• Both LSTMs and GRUs remain sequential-only and require backpropagation through time, producing linear training time that limits long-context scaling.Source: §2.2 · p. 3

2.3 Parallel Scan

• Transformers remove the recurrent bottleneck through parallel training but introduce quadratic sequence-length complexity, motivating parallel recurrent alternatives.New recurrent models commonly use parallel prefix scan to avoid BPTT-related limitations.Source: §2.3 · p. 3
• Parallel scan computes prefix recurrences such as vt = at ⊙vt−1 + bt in parallel when the input sequences of coefficients and increments are available.The method extends from scalar recurrences to vector-valued element-wise updates.Source: §2.3 · p. 3

3 Methodology

• The paper reformulates LSTM and GRU recurrences for parallel scan by removing gates’ dependencies on previous hidden states and simplifying output constraints.This produces minimal recurrent models that remain input-dependent while making recurrence inputs computable in parallel.Source: §3 · p. 3, §3.1.1 · p. 4, §3.1.2 · p. 4

3.1.1 Step 1: Drop previous state dependencies from gates

• Removing previous-state dependencies lets GRU coefficients and candidate terms be computed in parallel, while also eliminating the reset gate’s prior-state role.The paper notes empirical support for omitting such dependencies in models including xLSTM and Mamba, while acknowledging theoretical concerns.Source: §3.1.1 · p. 4

3.1.2 Step 2: Drop range restriction of candidate states

• Dropping tanh removes the range restriction on GRU candidate and hidden states after previous-state dependencies have been removed.Source: §3.1.2 · p. 4

3.1.3 minGRU

• minGRU combines the GRU simplifications into a model requiring O(2dhdx) parameters versus GRU’s O(3dh(dx + dh)).With state expansion α = 1, 2, 3, 4, minGRU uses approximately 33%, 22%, 17%, and 13% of GRU parameters, respectively.Source: §3.1.3 · pp. 4–5
• minGRU can be trained in parallel with parallel scan, bypassing backpropagation through time.Source: §3.1.3 · p. 5

3.2.1 Step 1: Drop previous state dependencies from gates

• LSTM simplification removes previous-state gate dependencies, tanh range restrictions, and the output gate, allowing the cell state and hidden state to collapse into one state.The resulting recurrence can normalize input and forget gates so state scale remains time-independent.Source: §3.2.1 · p. 5, §3.2.2 · p. 5, §3.2.3 · pp. 5–6

3.2.4 minLSTM

• minLSTM requires O(3dhdx) parameters versus LSTM’s O(4dh(dx + dh)), using approximately 38%, 25%, 19%, or 15% of LSTM parameters for α = 1, 2, 3, or 4.Source: §3.2.4 · p. 6
• minLSTM can be trained in parallel using parallel scan, bypassing backpropagation through time.Source: §3.2.4 · p. 6

4.1 Minimal LSTMs and GRUs are efficient

• At sequence length 4096, minGRU and minLSTM were 1324× and 1361× faster per training step than GRU and LSTM, respectively.At length 512, the corresponding speedups were 175× and 235×.Source: §4.1 · p. 7
• MinGRU and minLSTM use parallel scan for training, trading approximately 88% more memory than traditional GRU and LSTM models.Mamba uses 56% more memory than minGRU, while runtime is described as the practical bottleneck.Source: §4.1 · p. 7
• Adding layers substantially improves Selective Copying performance because outputs from earlier layers become time-dependent inputs to later layers.The reported comparison is provided in Table 1.Source: §4.1 · p. 8, Table 1 · p. 8
• MinGRU solves Selective Copying more consistently and with lower variance than minLSTM, which uses separate forget and input gates.The authors attribute the stability difference to minGRU controlling information discarding and addition with one update gate.Source: §4.1 · p. 8

4.2 Minimal RNNs perform surprisingly well

• On Selective Copying, minLSTM and minGRU solve the task comparably to Mamba’s S6 and outperform S4, H3, and Hyena.The baseline results are reported in Table 2, with only Mamba’s S6 among the listed baselines also solving the task.Source: §4.2 · p. 8, Table 2 · p. 8
• Across nine D4RL datasets, minLSTM and minGRU outperform Decision S4 and perform comparably with Decision Transformer, Aaren, and Decision Mamba.They outperform all listed baselines except Decision Mamba, where the difference is marginal.Source: §4.2 · p. 9, Table 3 · p. 9
• On Shakespeare language modelling, minGRU, minLSTM, Mamba, and Transformers achieved comparable test losses, while Transformers required approximately 2.5× more training steps than minGRU.The reported test losses were 1.548, 1.555, 1.575, and 1.547, respectively.Source: §4.2 · p. 9, Figure 2 · p. 9

State-Space Models (SSMs)

• State-space models such as S4 and Mamba combine recurrent-style inference with convolution-like parallel training, with Mamba introducing input-dependent transition matrices.S4 motivated later developments across language and audio applications.Source: §5 · p. 10

Recurrent Versions of Attention

• Recurrent attention models extend linear attention with gating, using either input-independent decay factors or input-dependent gates, while some reinterpret softmax attention as an RNN.These approaches form a separate direction within efficient recurrent sequence modelling.Source: §5 · p. 10

Parallelizable RNNs

• Parallelizable RNN research uses convolutions, parallel scan, or modified recurrent dependencies to retain efficient training while approximating traditional state interactions.Examples include GILR, linear gated RNNs, HGRN, xLSTM, and ternary-weight recurrent mixers; minGRU is equivalent to GILR without an activation function.Source: §5 · p. 10

6 Conclusion

• MinLSTMs and minGRUs use fewer parameters, support fully parallelizable training, and achieve competitive performance despite their simplicity.The authors present these minimal versions as a basis for reevaluating simpler recurrent architectures alongside newer models.Source: §6 · p. 10
• The work revisits LSTMs and GRUs, which dominated sequence modelling for two decades before Transformers rose to prominence.Source: §6 · pp. 10–11

Limitations

• Older 16 GB GPUs, compared with 80 GB A100s used for modern baselines, limited the scale of the evaluations.Gradient accumulation reduced the effective batch size by half for some tasks and significantly slowed training.Source: Limitations · p. 11
• The authors believe their conclusions are likely to generalize to larger-scale settings because the minimal models share fundamental similarities with recent recurrent models.This expectation is presented as a belief rather than a demonstrated result under larger-scale conditions.Source: Limitations · p. 11

A.1.1 minGRU: A Minimal GRU

• The implementations define sequential and parallel modes for minGRU, with both taking x and an initial hidden state and producing hidden-state sequences.Source: §A.1.1 · p. 15, §A.2.1 · p. 16

A.1.2 minLSTM: A Minimal LSTM

• The implementations define sequential and parallel modes for minLSTM with length independence scaling, using sequence inputs and an initial hidden state.Source: §A.1.2 · p. 15, §A.2.2 · p. 16

A.2.1 minGRU: A Minimal GRU

• The PyTorch listings expose separate sequential and parallel forward interfaces for minLSTM and minGRU.Source: §A.2.1 · p. 16, §A.2.2 · p. 16
• The minGRU recurrence combines the previous hidden state and candidate state using an input-dependent update gate.Source: §A.2.1 · p. 16

B Implementation Details: Log-Space Version (Additional Numerical Stability)

• During training, parallel modes avoid backpropagation through time, while sequential modes are used at inference.The log-space implementation is intended to improve numerical stability during these computations.Source: B Implementation Details: Log-Space… · p. 17, §B.2.1 · p. 17

B.1 Parallel Scan: Log-Space Implementation

• The log-space parallel scan takes log coefficients and log values as inputs and returns the hidden-state sequence.Source: §B.1 · p. 17

B.2.1 minGRU: A Minimal GRU

• For minGRU, the recurrence is expressed through coefficients 1−z_t and values z_t ⊙ h̃_t, then evaluated with a log-space parallel scan.A continuous activation ensures h̃_t is positive so its logarithm can be computed without complex numbers.Source: §B.2.1 · pp. 17–18

B.2.2 minLSTM: A Minimal LSTM

• For minLSTM, sigmoid-based input and forget gates are transformed into log-space coefficients before combining them in a parallel scan.The resulting derivation yields a parallel mode for efficient training.Source: §B.2.2 · pp. 18–19

B.3 PyTorch Code: Log-Space Version

• The PyTorch code provides log-space sequential and parallel implementations for both minGRU and length-scaled minLSTM.Source: §B.3 · p. 19, §B.3.1 · pp. 19–20, §B.3.2 · p. 20

C Detailed Experiment Setup

• The document also includes a detailed experiment-setup section after the log-space implementation material.Source: C Detailed Experiment Setup · p. 21

C.1 Datasets

• The datasets cover Selective Copying, Chomsky Hierarchy, Long Range Arena, language modeling, and continuous-control reinforcement learning.Long Range Arena includes Retrieval, ListOps, and G-Image; reinforcement learning uses HalfCheetah, Hopper, and Walker environments.Source: §C.1 · p. 21
• Chomsky Hierarchy models train on sequences up to length 40 and evaluate generalization to lengths between 40 and 256.Source: §C.1 · p. 21
• Long Range Arena evaluates long-range modeling with sequence lengths from 1024 to 4000 across Retrieval, ListOps, and G-Image.Source: §C.1 · p. 21

C.2 Architecture

• The architecture uses no additional components for Selective Copying, while Chomsky Hierarchy adds a kernel-size-4 temporal convolution before the minimal RNN.Source: §C.2 · pp. 21–22
• For complex tasks such as language modeling and Long Range Arena, the architecture adds a convolutional layer and an MLP.Source: §C.2 · pp. 21–22

C.3 Hyperparameters and general experimental details

• The general setup uses benchmark-specific optimizers, learning rates, batch sizes, dropout, and model depths rather than one shared configuration.Source: §C.3 · p. 22
• Training configurations vary by benchmark, including 250k steps for Long Range Arena, 500k steps for Chomsky Hierarchy, and 5k steps for language modeling.Source: §C.3 · p. 22

D.1 Chomsky Hierarchy + Long Range Arena

• A kernel-size-4 convolution is prepended before the recurrent unit, following a design used in models such as Mamba and xLSTM[1:0].The authors report that this convolution also helped their minimal RNNs.Source: §D.1 · p. 22
• The experiments compare Minimal RNNs with RWKV, Mamba, and xLSTM[1:0], averaging results over 3 seeds and using baseline accuracies from the xLSTM paper.Source: §D.1 · p. 23
• Minimal RNNs achieve competitive performance with state-of-the-art models such as Mamba and xLSTM across all Chomsky Hierarchy and Long Range Arena tasks.The comparison also reports better performance than Retention, Hyena, RWKV, and Llama.Source: §D.1 · p. 23

D.2 Inference Runtime Comparison

• The runtime comparison varies batch sizes from 8 to 64 and sequence lengths up to 2048, averaging inference speed across 50 runs with context tokens included.Traditional GRU and LSTM process context sequentially, whereas minGRU, minLSTM, and Mamba benefit from parallel processing.Source: §D.2 · p. 23, Figure 3 · p. 25
• MinGRU is 6.6%, 4.1%, 4.9%, and 2.9% faster than Mamba at batch sizes 8, 16, 32, and 64, respectively.MinLSTM is 3.6%, 2.9%, 0%, and 1.3% slower than Mamba at those batch sizes.Source: §D.2 · p. 23
• At batch size 64, minGRU and minLSTM are 19.6% and 41.5% faster than GRU and LSTM, respectively.Source: §D.2 · p. 23

D.3 Architecture Ablation

• Adding a convolutional layer and an MLP improves minLSTM performance on the ListOps dataset.The ablation averages results over 3 seeds.Source: §D.3 · p. 23, Table 6 · p. 25

D.4 Initialization Analyses

• Increasing the minLSTM forget-gate bias promotes earlier information retention and leads to faster convergence and greater training stability.Source: §D.4 · pp. 23–24, Figure 5 · p. 27

E Additional Related Work

• MinLSTM and minGRU use a parallel-scan recurrence, while hardware-efficient scan methods such as chunking can also accelerate their training.The implementations in this work use plain PyTorch.Source: E Additional Related Work · pp. 24–25