The RNN Homunculus: Learning to Gate Long-Term Memory

1 INTRODUCTION

• RNNs offer linear-time inference with fixed-size recurrent states, while self-attention has quadratic inference cost in sequence length.The introduction frames recurrent models as a complementary efficiency trade-off for long-range sequence modeling.Source: §1 · p. 1
• Gate values that protect long-range gradients must themselves be learned through recurrent computation subject to long-range gradient decay.The usual additive-copy-path explanation assumes useful gate values, leaving their learning unexplained.Source: §1 · pp. 1–2
• The paper hypothesizes that short-range sub-tasks let the Gate-RNN learn first, enabling the Main-RNN to use the resulting memory path on long-range tasks.Controlled experiments add short-dependency samples to test whether learning generalizes to long-range dependencies.Source: §1 · p. 2
• Gate learning in one class can benefit long-range dependency learning in different classes.The paper evaluates whether learning shared across classes transfers across class labels.Source: §1 · p. 2
• Short-dependency training support enables gated RNNs to close the accuracy gap with Mamba and xLSTM.The paper reports that gated RNNs can match or exceed their test accuracy.Source: §1 · p. 2

The Vanishing Gradient Problem

• Vanishing gradients arise when products of recurrent Jacobians decay with temporal distance, making long-range contributions to learning negligible relative to short-range ones.This analysis motivates additive state paths intended to preserve information and gradients over longer intervals.Source: §2 · p. 3

Gated Recurrent Networks

• LSTM and GRU additive copy paths can preserve prior state, but their effectiveness depends on learned forget- or update-gate values.Prior empirical comparisons find advantages over vanilla RNNs, with relative GRU and LSTM accuracy varying across tasks.Source: §2 · p. 3

Modern Recurrent Architectures

• Modern recurrent models vary in gate design: minGRU and minLSTM use input-only gates, Mamba uses input-dependent state-space parameters, and xLSTM adds exponential gating and modified memory structures.The paper compares these models with gated RNNs to examine how short-dependency samples affect long-dependency task accuracy.Source: §2 · p. 3

What Gated Recurrent Networks Learn

• Prior analyses of trained LSTMs identify long-distance agreement, filler–gap tracking, and units associated with grammatical or structural information.This paper complements those studies by examining learning signals to gates and how short-dependency samples support long-range learning.Source: §2 · p. 3

3 THE GATE LEARNING PROBLEM

• GRU and LSTM gates can preserve state gradients, but learning useful gate values still depends on task-loss gradients reaching the relevant recurrent state.For an early gate, that signal passes through a long product of recurrent Jacobians; sigmoid saturation can attenuate it further.Source: §3 · p. 4

3.1 EXTENSION TO MORE GENERAL GATING

• Even when gates are computed from current inputs alone, their learning signals can decay exponentially with temporal distance, so removing hidden-state dependence does not solve gate learning.This bound applies when later gate values remain below 1.Source: §3.1 · p. 4

3.2 LEARNING FROM SHORT DEPENDENCIES

• Short-dependency samples send gate-learning signals through fewer transformations, and shared gates learned at one position are applied across the sequence.The authors hypothesize that these samples help gates learn what to write into memory and ignore before selective updates must preserve information over longer intervals.Source: §3.2 · p. 5

4 EXPERIMENTS

• Controlled temporal-XOR sequences keep the label rule and sequence length fixed while moving two informative tokens among non-informative tokens; four classes test transfer across token groups.Long-dependency samples place informative tokens across the sequence, while short-dependency samples place them within the final min(10, T) positions; validation and test samples use hard-only placement.Source: §4 · p. 5

4.1 HOW DO GATES LEARN LONG RANGE DEPENDENCIES?

• Hard-only training drives GRUs and LSTMs to 50% on long sequences, while short-dependency samples sustain approximately 90% or higher hard-test accuracy by teaching shared gates when to write or preserve memory.The vanilla RNN stays close to its hard-only accuracy despite using a larger hidden state in mixed training.Source: Figure 3 · p. 6, §4.1 · p. 6

4.2 DOES GATE LEARNING TRANSFER ACROSS CLASSES?

• Support only for group {2, 3} raises its accuracy above approximately 90% and the unsupported group to 70–85%, but direct support helps more.The partial transfer is consistent with shared memory-preservation behavior, while writing relevant information and mapping it to an output remain group-specific.Source: Figure 4 · p. 7, §4.2 · pp. 7–8

4.3 LEARNING DYNAMICS AND GRADIENT PROPAGATION

• In mixed GRU training, hard-validation accuracy reaches 100% around epochs 55–60 as gate learning enables the Main-RNN to exploit long-range samples.Early-position gradient maxima are roughly nine orders of magnitude lower under hard-only training than at final positions, but remain within roughly one order of magnitude across positions in the successful mixed run.Source: Figure 5 · p. 8, §4.3 · p. 8

4.4 EXTENSION TO MODERN GATED RNNS AND STATE-SPACE MODELS

• With short-dependency support, minGRU and minLSTM reach 100% at every tested length, and gated models match or exceed Mamba and xLSTM on mixed data.Under hard-only training, Mamba and xLSTM maintain mean accuracies above 80% throughout the tested range, while LSTM, minGRU, and minLSTM approach chance as sequence length increases.Source: Figure 6 · p. 9, §4.4 · p. 9

5 DISCUSSION

• Short-dependency samples help GRUs and LSTMs learn long-range tasks, with partial transfer across informative tokens and output labels; minGRU and minLSTM match or exceed Mamba and xLSTM test accuracy.The authors also report that recurrent computation can attenuate gate-learning signals, even for input-only gates.Source: §5 · p. 9
• The probes do not isolate the causal roles of gradient propagation and selective updates, and comparable short-dependency support in large-scale real-world models remains an open challenge.The authors also leave unexplained why Mamba and xLSTM learn more reliably from hard samples under the tested configurations.Source: §5 · p. 9
• Training-data dependency structure offers a practical way to improve long-range learning in existing recurrent architectures.Exposing the same task over shorter dependencies can help shared gates acquire memory behavior useful over longer intervals.Source: §5 · p. 9
• The authors report using ChatGPT for experiment design, result interpretation, derivation checks, code assistance, literature search, and an initial draft, with manuscript revision and validation performed by the authors.The authors state that they take full responsibility for the final manuscript, claims, references, and code.Source: AI USE STATEMENT · p. 10
• The reproducibility statement locates derivations, task and evaluation details, model configurations, optimization procedures, and diagnostic measurements across the main text and appendices.Appendix A.2 documents stopping criteria, checkpoint selection, and seeds for model and data.Source: REPRODUCIBILITY STATEMENT · p. 10

A.1 MODEL CONFIGURATIONS

• All classifiers use learned 8-dimensional token embeddings, one recurrent layer or sequence-model block, and a linear head on the final output.The head has two outputs for binary XOR and four for multi-class XOR; recurrent states reset for every sequence.Source: §A.1 · p. 13
• Vanilla RNNs, GRUs, and LSTMs use explicit sequential updates; the vanilla RNN uses tanh, and these models have zero initial states with no dropout or normalization.The excerpt specifies uniform initialization for recurrent weights and biases, but its bound is incomplete.Source: §A.1 · p. 13
• The minGRU and minLSTM use input-only gates and linear input-only candidates, with minLSTM normalizing its sigmoid gates by their sum in the log domain.Both use zero initial states and the same uniform parameter initialization as the basic recurrent models.Source: §A.1 · p. 13
• The comparison uses a single small Mamba-1 block and the official xLSTM implementation configured with one sLSTM block, width 10, and one head.Mamba uses state dimension 2 and convolution kernel size 4; xLSTM uses a causal convolution with kernel size 4.Source: §A.1 · p. 13

A.2 OPTIMIZATION AND EVALUATION

• Models train with cross-entropy and AdamW, using batch size 500, weight decay 10^-4, and learning rates 5 × 10^-3 for binary XOR and 10^-2 for multi-class XOR.Training uses full backpropagation through the sequence, without gradient clipping or a learning-rate schedule, and stops by 500 epochs or early validation criterion.Source: §A.2 · pp. 13–14
• Each sweep setting uses 25 combinations of model and data seeds, and reported test accuracy is the mean and standard deviation across runs.Validation and test data use separate generators; multi-class group accuracies restrict original four-class predictions to the relevant informative-token group.Source: §A.2 · p. 14

B ADDITIONAL DIAGNOSTIC PROBES

• The additional probes use binary XOR at T = 100, with short-dependency placement in 40% of mixed-training samples and hard-only validation.Every three epochs, one shuffled training batch is used to average gate values and hidden-state changes by token category.Source: B ADDITIONAL DIAGNOSTIC PROBES · p. 14

B.1 GRU: GATE VALUES AND HIDDEN-STATE CHANGES

• Mixed training produces higher update-gate values for non-informative tokens and lower values for informative tokens, whereas hard-only gate values are not clearly separated.The resulting policy copies across non-informative tokens and writes when informative tokens appear.Source: §B.1 · p. 14, Figure B.1 · p. 15
• Under hard-only training, hidden-state changes increase across all token categories; in the mixed run, non-informative-token changes fall while informative-token updates remain larger.The decrease in non-informative-token changes occurs around the transition to successful hard-validation performance.Source: §B.1 · p. 14, Figure B.2 · p. 15

• Mixed training reaches high hard-validation accuracy after initial improvement, while hard-only validation stays near chance.Source: §B.2 · p. 16, Figure B.3 · p. 16
• The mixed run sends stronger gradients to early hidden states than the hard-only run.Source: §B.2 · p. 16, Figure B.4 · p. 16
• Mixed training increases forget-gate values on non-informative tokens and yields smaller hidden-state updates there than at informative tokens.Forget-gate averages alone do not establish state preservation; the full update also depends on the input gate, candidate, and output gate.Source: §B.2 · p. 16, Figure B.5 · p. 17, Figure B.6 · p. 17

• Vanilla RNNs do not reach high hard-validation accuracy in the illustrated runs under either hard-only or mixed training.Source: §B.3 · p. 18, Figure B.7 · p. 18
• Their gradients remain much smaller at early positions than near the final prediction under both training conditions.Source: §B.3 · p. 18, Figure B.8 · p. 18
• Non-informative tokens continue to produce substantial hidden-state changes during mixed training, without selective updates like those in gated models.Source: §B.3 · p. 18, Figure B.9 · p. 19

C.1 SENSITIVITY TO THE SHORT-DEPENDENCY WINDOW SIZE

• At T = 100 and peasy = 40%, intermediate suffix windows perform best; both extremes hurt, and the main-experiment window 10 lies in that regime.Very small windows provide limited non-informative context, while very large windows increase dependency length.Source: §C.1 · pp. 19–20, Figure C.1 · p. 19

C.2 SYMMETRY CHECK FOR THE MULTI-CLASS XOR SETTING

• Treating {0, 1} improves its accuracy reliably and also raises untreated {2, 3} over hard-only, but transfer is less stable and below the fully treated reference.Validation and test samples remain hard for both groups.Source: §C.2 · p. 20, Figure C.2 · p. 20