Source-linked AI summary
How Temporal Correlations Shape Memory in Linear Recurrent Neural Networks
Arnol Manuel Fokam, Fasseu Sieyondji Akpevwoghene, Edem Fiifi Dawson
TL;DR
Existing theory largely treats inputs as temporally independent, leaving unclear how correlation shapes memory formation. This paper solves linear-RNN learning dynamics for correlated inputs and finds that correlation changes the cost of retaining the past, altering learning trajectories, settled memory, and the network structure needed for zero error.
Problem
Existing theory relies on temporally independent inputs, leaving the interaction between input predictability and memory formation quantitatively unresolved.
Method
The paper analytically extends a linear-RNN learning framework to temporally correlated inputs and derives the resulting learning dynamics and task-optimal network structure.
Results
Correlation reshapes learning through the memory penalty: memory can overshoot and be removed, positive correlation increases its cost, and a spare hidden dimension learns the feedthrough required for zero error.
Takeaways & Limitations
A dataset’s temporal correlation and task demand can predict whether training retains the past or instead produces change detection, while exact solutions may require recurrence alongside feedthrough.
Takeaways & Limitations
The theory assumes shared task singular vectors, whitened within-timestep features, and stationary temporal covariance.
Abstract
from arXiv · showhide
The linear recurrent neural network (LRNN) is a simple model for studying how much memory a network builds up as it trains. For uncorrelated inputs, earlier work found that training itself settles the network between keeping the past and reacting only to the present. Real sequences are correlated, and we solve the learning dynamics exactly for correlated inputs. In the solution, keeping the past carries a cost. The whole effect of correlation lands on that cost. This cost reduces to the earlier one when inputs are uncorrelated and grows once they are positively correlated. Three findings follow. (1) Correlation reshapes the course of learning, not only its end. Memory builds, overshoots, and is partly removed, and the settled network keeps less of the past. (2) Memory switches off at a threshold set by one number, how much each input resembles the one just before it. Neither sequence length nor longer-range correlation moves this threshold. Memory is worth keeping only when the task needs the previous input more than the current input already supplies it through correlation with the past. (3) The best network changes too. Zero error demands a feedthrough, a path that passes the current input straight to the network's output and remembers nothing, and training builds it unprompted when given one spare hidden dimension. Our work turns one property of the input into a prediction of whether a network learns memory and explains why correlated data turns recurrent networks into change detectors.
Introduction
This paper extends linear-RNN learning theory from temporally independent to correlated inputs, using exact analysis to connect temporal structure with learned memory. It models sequence-to-one regression through a recurrent hidden state trained by gradient flow under structured task and input-covariance assumptions.
- Introduction: Existing theory assumes temporally independent inputs and therefore misses how input predictability interacts with memory formation.As predictability increases, retaining past information becomes less beneficial, creating a trade-off between external statistical structure and internal memory.
- Introduction: The paper extends an earlier linear-RNN framework by deriving exact learning dynamics for temporally correlated inputs.The linear RNN preserves temporal information propagation while remaining analytically tractable.
- The Task: The task is sequence-to-one regression: each trajectory supplies T inputs sequentially and one target produced after the final input.The task is characterized by input-target correlations at each step and input-input correlations across timestep pairs.
- The Task: The analysis assumes shared singular directions for task correlations, whitened features within each timestep, and stationary temporal covariance across timesteps.These assumptions separate independent task directions from temporal input structure while keeping the dynamics solvable in closed form.
- Model and Training: The LRNN reads inputs into a hidden state, propagates that state recurrently, and maps the final state to the output while minimizing average squared error.An input arriving earlier passes through more powers of the recurrent matrix before reaching the output.
- Model and Training: Gradient flow updates the three weight matrices over training time, with aligned initialization making the learning dynamics analytically tractable.Training time is distinct from sequence position, and the hidden basis is aligned with the task’s singular directions.
Findings
Correlated inputs reshape both the cost and dynamics of memory: training may overshoot or reverse the memory dial, and the settled outcome is determined by task demand relative to recent input correlation. Zero-error solutions additionally require a feedthrough component that training discovers with one spare hidden dimension.
- Correlation Reshapes the Learning Dynamics: Correlation changes only the effective regularization term, increasing the cost of positive memory through covariance cross terms.The data-driven and learning-rule components remain unchanged; correlation alters the score assigned to the network output.
- Correlation Decides When Memory Matters: Memory is retained only when task demand exceeds the correlation between the two most recent inputs; sequence length and longer-range covariance do not move this boundary.At the boundary, b settles at zero; below it, the network detects change rather than storing the past.
- Correlation Decides When Memory Matters: For the geometric task with AR(1) inputs, the boundary is λ = ρ, separating positive memory, zero memory, and negative-memory change detection.Figure 2 shows the same diagonal boundary across a 4 × 6 grid, with every run reaching its energy minimum.
- Correlation Demands a Feedthrough Term: Zero error requires a feedthrough plus a recurrent tail, but a single hidden dimension cannot represent both components off the boundary.The feedthrough passes the current input directly, while the recurrent mode stores the residual temporal dependence.
- Correlation Demands a Feedthrough Term: With one spare hidden dimension, gradient flow independently builds the feedthrough: one dial settles at task demand 0.6 and the other at zero in all one hundred runs.At width one, the lone mode instead settles into increasingly strong change detection as correlation rises.
Related Work
The paper extends exact learning-dynamics analyses of linear networks to correlated inputs, while connecting its recurrence to estimation theory and efficient coding. Its distinct contribution is showing when gradient descent collapses memory and selects whitening-like behavior under recurrent capacity constraints.
- Learning dynamics of linear networks: The work builds on exact-solution analyses where learning decouples along singular directions into independent scalar modes.This framework has also been extended to generalization, rich and lazy regimes, and implicit low-rank bias.
- Memory capacity of recurrent networks: Memory-capacity research measures retention in fixed or linear recurrent systems, whereas this paper studies how training selects memory under correlated inputs.Its focus is learned dynamics rather than capacity bounds alone.
- Optimal estimation under correlated inputs: The derived optimal readout is a Wiener filter or generalized-least-squares estimator, but the paper studies its emergence through gradient descent under recurrent capacity constraints.Estimation theory does not ask which recurrence learning selects or when memory collapses.
- Whitening in the brain: The learned rule resembles efficient coding: subtract predictable input structure and retain what is new.The paper connects this whitening behavior to a measurable recent-step correlation that predicts retained memory.
Conclusion
Temporal correlation enters learning as a cost for retaining the past, making memory predictable from the data and shifting networks toward change detection. Exact task solutions require recurrence alongside a feedthrough, which training can recruit automatically.
- Correlation acts as a price for holding onto the past, and memory remains only when prior-input demand exceeds input correlation.The comparison can be read from the data before training.
- Below the memory boundary, the network changes its computation and becomes a change detector rather than merely storing less.
- Exact task solutions combine recurrence with a feedthrough, and gradient descent recruits this component when an extra hidden dimension is available.
A. Mode Decoupling under Correlated Inputs
Under isotropic temporal covariance, correlations span timesteps but not task modes, so the loss and gradient flow decouple into independent mode-wise dynamics. Alignment is preserved throughout training, reducing the full matrix problem to one diagonal mode at a time.
- The recurrent readout weights older inputs by powers of the recurrent mode, while the newest input arrives without recurrent damping.
- Temporal covariance contributes across timesteps while isotropy prevents correlations across modes, preserving mode decoupling.The temporal part is Cij, while the mode part is δαβ.
- The input-target statistic is diagonal in the same task modes, with sα,i representing mode α’s demand at timestep i.
- The rotated loss separates into one independent problem per mode because each output coordinate depends only on its matching mode.
- Gradient flow preserves aligned, mode-diagonal parameter forms, so cross-mode components are never created during training.
B. Memory Cost and Per-Mode Learning Dynamics
For each mode, temporal correlation changes the memory cost while leaving the data-driven demand unchanged. After optimizing the gain, learning adjusts the memory dial by comparing demand against this correlation-dependent cost, subject to stable recurrent dynamics and finite-sample estimation error.
- The derivation assumes positive-definite covariance and stable modes |b| < 1, while finite-sample dynamics incur O(P^-1/2) second-moment estimation error.
- Correlation changes only the memory-cost term, while the data-driven term remains determined by target-input correlations.All new covariance dependence enters through the expected squared recurrent output.
- The memory cost is the positive quadratic form v⊤Cv, replacing the white-input cost in the parameter-dependent energy.
- Positive off-diagonal correlations add a surcharge, making the same positive memory strictly more expensive than under white inputs.
- After profiling out gain g = ca, the memory dial optimizes memory-weighted task demand against the correlation-dependent regularization cost.
- Learning follows gradient flow on the profiled energy, with the dial increasing when demand slope exceeds overlap-cost slope.
C. Lag-One Transition and Tail Invariance
The memory boundary is set exactly by the task’s demand for the previous input versus the input’s lag-one overlap. Sequence length and covariance lags two and above do not move this transition, while white inputs recover the earlier zero threshold.
- Boundary condition: Memory switches on when the task demand ratio exceeds the input overlap ratio, giving λ⋆ = ρ for geometric tasks with AR(1) inputs.The transition compares sT−1/sT with C1/C0; for the geometric task these are λ and ρ.
- Change detection: When task demand is below input overlap, the unconstrained optimum moves toward negative b, corresponding to change detection rather than positive recurrence.On the constrained nonnegative axis, the feedforward endpoint remains stable when the drift is negative.
- Scope of the boundary: The endpoint calculation is exact locally, while global agreement is confirmed numerically in the AR(1)-like regime but can fail for strongly periodic or highly persistent covariances.Those exceptional covariances can produce a distant minimum that overtakes the endpoint.
- Tail invariance: The boundary depends only on newest-input variance and lag-one covariance, so every covariance lag d ≥2 is invisible to the transition.Longer-range covariance can still affect the settled memory magnitude away from the boundary.
- Tail invariance: Sequence length T does not shift the transition because the boundary reads the task and covariance through ratios that remain unchanged with T.Lengthening the sequence adds only higher-order coefficients to the energy expansion.
- White-input limit: For white inputs, C1 = 0 and the threshold returns to λ⋆ = 0, matching the earlier uncorrelated-input cost.With no lag-one overlap, any demand for the second-newest input switches memory on.
D. Optimality Gap and Feedthrough Capacity
Correlated inputs can make a single recurrent exponential insufficient: the optimal response combines a geometric tail with a feedthrough component. Adding one eigenvalue-zero hidden dimension per task mode closes this representation gap, and gradient descent recruits it without explicit prompting.
- Optimality gap: Correlation opens a nonnegative optimality gap because the unrestricted Wiener/GLS response need not be a single exponential.The gap vanishes only when C−1sα is proportional to a single exponential response.
- Optimality gap: For AR(1) inputs and geometric tasks, the optimal filter combines a newest-step spike with a geometric tail.The ratio of the step-(T −1) tap to the newest-step tap is λα −ρ, and a single ray represents both components only at ρ = 0 or ρ = λα.
- Feedthrough capacity: At hidden width Nh = 2N, training reaches the Wiener value −2.2772 with an optimality gap of 3.3 × 10−9, compared with the width-N single-pole ceiling −1.3176.The learned spectrum contains one pole per task rate and one zero-eigenvalue feedthrough channel per mode.
- Feedthrough capacity: For one task mode, width two closes the single-ray gap to 3.29 × 10−9 in all twenty runs, while extra width does not improve recruitment.Width one remains at a gap of 2.85 × 10−1 in twenty of twenty runs.
- General covariance families: Beyond AR(1), covariance structure determines the correction around the task-rate geometric tail, and finite-order families require feedthrough, poles, or seasonal components accordingly.The selected components close the single-ray gap below 10−6 across reported finite-order families.
- Feedthrough capacity: A single recurrent mode cannot represent the correlated optimum, whereas one eigenvalue-zero feedthrough dimension per task mode can repair the gap.The feedthrough passes the current input directly to the output while adding no recurrent memory.
E. Real-Data Validation
The paper tests its threshold and representation claims on real and synthetic covariance structures. Across nineteen series, learned crossings match the measured newest-step thresholds within the stated bisection bias, while scope-condition cases exhibit first-order transitions instead.
- Protocol: The real-data protocol standardizes or differences series, uses overlapping windows of length T = 20, and estimates temporal covariance from each window matrix.Series are sampled at their natural resolutions across finance, weather, hydrology, air quality, macroeconomics, sunspots, CO2, Niño 3.4, and EEG.
- Threshold validation: Across nineteen series, thresholds span 0.1438 to 0.9828, and every learned crossing lies within 1.5 × 10−4 of its measured threshold.The measured threshold is the correlation of the two newest covariance steps.
- Threshold validation: The newest-step threshold, rather than average lag-one correlation, predicts the crossing: US inflation has lag-one value 0.5941 but threshold 0.6152.Its crossing follows the newest-step threshold under the nonstationary formulation.
- Invariance tests: Synthetic covariances sharing threshold 0.6 show crossing movement of at most 6.0 × 10−5 when sequence length and longer-range covariance tails vary.This supports the predicted invariance to window length and correlations beyond the newest pair.
- Negative thresholds: Negatively correlated newest samples produce negative thresholds, so no memory transition occurs within λ ∈ (0, 1); both corresponding series already show positive settled dials at λ = 0.01.The measured thresholds are −0.154 for daily S&P returns and −0.041 for daily Bitcoin returns.
- Scope condition: Strong periodicity or extreme persistence can produce first-order transitions in which a distant energy minimum overtakes the endpoint before the local memory-release condition fires.This is a scope condition on the complete, continuous transition described for AR(1)-like covariances.