PaperlayerView exact revision on arXiv

Source-linked AI summary

How Temporal Correlations Shape Memory in Linear Recurrent Neural Networks

Arnol Manuel Fokam, Fasseu Sieyondji Akpevwoghene, Edem Fiifi Dawson

arXiv:2609.00420v1cs.LG

TL;DR

Existing theory largely treats inputs as temporally independent, leaving unclear how correlation shapes memory formation. This paper solves linear-RNN learning dynamics for correlated inputs and finds that correlation changes the cost of retaining the past, altering learning trajectories, settled memory, and the network structure needed for zero error.

Abstract

from arXiv · show

The linear recurrent neural network (LRNN) is a simple model for studying how much memory a network builds up as it trains. For uncorrelated inputs, earlier work found that training itself settles the network between keeping the past and reacting only to the present. Real sequences are correlated, and we solve the learning dynamics exactly for correlated inputs. In the solution, keeping the past carries a cost. The whole effect of correlation lands on that cost. This cost reduces to the earlier one when inputs are uncorrelated and grows once they are positively correlated. Three findings follow. (1) Correlation reshapes the course of learning, not only its end. Memory builds, overshoots, and is partly removed, and the settled network keeps less of the past. (2) Memory switches off at a threshold set by one number, how much each input resembles the one just before it. Neither sequence length nor longer-range correlation moves this threshold. Memory is worth keeping only when the task needs the previous input more than the current input already supplies it through correlation with the past. (3) The best network changes too. Zero error demands a feedthrough, a path that passes the current input straight to the network's output and remembers nothing, and training builds it unprompted when given one spare hidden dimension. Our work turns one property of the input into a prediction of whether a network learns memory and explains why correlated data turns recurrent networks into change detectors.

Introduction

This paper extends linear-RNN learning theory from temporally independent to correlated inputs, using exact analysis to connect temporal structure with learned memory. It models sequence-to-one regression through a recurrent hidden state trained by gradient flow under structured task and input-covariance assumptions.

Introduction

  • Existing theory assumes temporally independent inputs and therefore misses how input predictability interacts with memory formation.As predictability increases, retaining past information becomes less beneficial, creating a trade-off between external statistical structure and internal memory.Source: Introduction · p. 1
  • The paper extends an earlier linear-RNN framework by deriving exact learning dynamics for temporally correlated inputs.The linear RNN preserves temporal information propagation while remaining analytically tractable.Source: Introduction · p. 1

The Task

  • The task is sequence-to-one regression: each trajectory supplies T inputs sequentially and one target produced after the final input.The task is characterized by input-target correlations at each step and input-input correlations across timestep pairs.Source: The Task · pp. 1–2
  • The analysis assumes shared singular directions for task correlations, whitened features within each timestep, and stationary temporal covariance across timesteps.These assumptions separate independent task directions from temporal input structure while keeping the dynamics solvable in closed form.Source: The Task · p. 2, The Objective Function · p. 2

Model and Training

  • The LRNN reads inputs into a hidden state, propagates that state recurrently, and maps the final state to the output while minimizing average squared error.An input arriving earlier passes through more powers of the recurrent matrix before reaching the output.Source: The Architecture · p. 2, The Objective Function · p. 2
  • Gradient flow updates the three weight matrices over training time, with aligned initialization making the learning dynamics analytically tractable.Training time is distinct from sequence position, and the hidden basis is aligned with the task’s singular directions.Source: The Objective Function · p. 2

Findings

Correlated inputs reshape both the cost and dynamics of memory: training may overshoot or reverse the memory dial, and the settled outcome is determined by task demand relative to recent input correlation. Zero-error solutions additionally require a feedthrough component that training discovers with one spare hidden dimension.

Correlation Reshapes the Learning Dynamics

  • Correlation changes only the effective regularization term, increasing the cost of positive memory through covariance cross terms.The data-driven and learning-rule components remain unchanged; correlation alters the score assigned to the network output.Source: Correlation Reshapes the Learning… · p. 4

Correlation Decides When Memory Matters

  • Memory is retained only when task demand exceeds the correlation between the two most recent inputs; sequence length and longer-range covariance do not move this boundary.At the boundary, b settles at zero; below it, the network detects change rather than storing the past.Source: Correlation Decides When Memory Matters · pp. 4–5
  • For the geometric task with AR(1) inputs, the boundary is λ = ρ, separating positive memory, zero memory, and negative-memory change detection.Figure 2 shows the same diagonal boundary across a 4 × 6 grid, with every run reaching its energy minimum.Source: Correlation Decides When Memory Matters · p. 4

Correlation Demands a Feedthrough Term

  • Zero error requires a feedthrough plus a recurrent tail, but a single hidden dimension cannot represent both components off the boundary.The feedthrough passes the current input directly, while the recurrent mode stores the residual temporal dependence.Source: Correlation Demands a Feedthrough Term · pp. 5–6
  • With one spare hidden dimension, gradient flow independently builds the feedthrough: one dial settles at task demand 0.6 and the other at zero in all one hundred runs.At width one, the lone mode instead settles into increasingly strong change detection as correlation rises.Source: Correlation Demands a Feedthrough Term · pp. 6–7

Related Work

The paper extends exact learning-dynamics analyses of linear networks to correlated inputs, while connecting its recurrence to estimation theory and efficient coding. Its distinct contribution is showing when gradient descent collapses memory and selects whitening-like behavior under recurrent capacity constraints.

Learning dynamics of linear networks

  • The work builds on exact-solution analyses where learning decouples along singular directions into independent scalar modes.This framework has also been extended to generalization, rich and lazy regimes, and implicit low-rank bias.Source: Related Work · p. 7

Memory capacity of recurrent networks

  • Memory-capacity research measures retention in fixed or linear recurrent systems, whereas this paper studies how training selects memory under correlated inputs.Its focus is learned dynamics rather than capacity bounds alone.Source: Related Work · p. 7

Optimal estimation under correlated inputs

  • The derived optimal readout is a Wiener filter or generalized-least-squares estimator, but the paper studies its emergence through gradient descent under recurrent capacity constraints.Estimation theory does not ask which recurrence learning selects or when memory collapses.Source: Related Work · p. 7

Whitening in the brain

  • The learned rule resembles efficient coding: subtract predictable input structure and retain what is new.The paper connects this whitening behavior to a measurable recent-step correlation that predicts retained memory.Source: Related Work · p. 7

Conclusion

Temporal correlation enters learning as a cost for retaining the past, making memory predictable from the data and shifting networks toward change detection. Exact task solutions require recurrence alongside a feedthrough, which training can recruit automatically.

  • Correlation acts as a price for holding onto the past, and memory remains only when prior-input demand exceeds input correlation.The comparison can be read from the data before training.Source: Conclusion · p. 7
  • Below the memory boundary, the network changes its computation and becomes a change detector rather than merely storing less.Source: Conclusion · p. 7
  • Exact task solutions combine recurrence with a feedthrough, and gradient descent recruits this component when an extra hidden dimension is available.Source: Conclusion · p. 7

A. Mode Decoupling under Correlated Inputs

Under isotropic temporal covariance, correlations span timesteps but not task modes, so the loss and gradient flow decouple into independent mode-wise dynamics. Alignment is preserved throughout training, reducing the full matrix problem to one diagonal mode at a time.

B. Memory Cost and Per-Mode Learning Dynamics

For each mode, temporal correlation changes the memory cost while leaving the data-driven demand unchanged. After optimizing the gain, learning adjusts the memory dial by comparing demand against this correlation-dependent cost, subject to stable recurrent dynamics and finite-sample estimation error.

C. Lag-One Transition and Tail Invariance

The memory boundary is set exactly by the task’s demand for the previous input versus the input’s lag-one overlap. Sequence length and covariance lags two and above do not move this transition, while white inputs recover the earlier zero threshold.

Boundary condition

  • Memory switches on when the task demand ratio exceeds the input overlap ratio, giving λ⋆ = ρ for geometric tasks with AR(1) inputs.The transition compares sT−1/sT with C1/C0; for the geometric task these are λ and ρ.Source: C. Lag-One Transition and Tail… · pp. 12, 14

Change detection

  • When task demand is below input overlap, the unconstrained optimum moves toward negative b, corresponding to change detection rather than positive recurrence.On the constrained nonnegative axis, the feedforward endpoint remains stable when the drift is negative.Source: C. Lag-One Transition and Tail… · pp. 14–15

Scope of the boundary

  • The endpoint calculation is exact locally, while global agreement is confirmed numerically in the AR(1)-like regime but can fail for strongly periodic or highly persistent covariances.Those exceptional covariances can produce a distant minimum that overtakes the endpoint.Source: C. Lag-One Transition and Tail… · p. 15

Tail invariance

  • The boundary depends only on newest-input variance and lag-one covariance, so every covariance lag d ≥2 is invisible to the transition.Longer-range covariance can still affect the settled memory magnitude away from the boundary.Source: C. Lag-One Transition and Tail… · p. 15
  • Sequence length T does not shift the transition because the boundary reads the task and covariance through ratios that remain unchanged with T.Lengthening the sequence adds only higher-order coefficients to the energy expansion.Source: C. Lag-One Transition and Tail… · p. 15

White-input limit

  • For white inputs, C1 = 0 and the threshold returns to λ⋆ = 0, matching the earlier uncorrelated-input cost.With no lag-one overlap, any demand for the second-newest input switches memory on.Source: C. Lag-One Transition and Tail… · p. 15

D. Optimality Gap and Feedthrough Capacity

Correlated inputs can make a single recurrent exponential insufficient: the optimal response combines a geometric tail with a feedthrough component. Adding one eigenvalue-zero hidden dimension per task mode closes this representation gap, and gradient descent recruits it without explicit prompting.

Optimality gap

  • Correlation opens a nonnegative optimality gap because the unrestricted Wiener/GLS response need not be a single exponential.The gap vanishes only when C−1sα is proportional to a single exponential response.Source: D. Optimality Gap and Feedthrough… · pp. 16–17
  • For AR(1) inputs and geometric tasks, the optimal filter combines a newest-step spike with a geometric tail.The ratio of the step-(T −1) tap to the newest-step tap is λα −ρ, and a single ray represents both components only at ρ = 0 or ρ = λα.Source: D. Optimality Gap and Feedthrough… · p. 17

Feedthrough capacity

  • At hidden width Nh = 2N, training reaches the Wiener value −2.2772 with an optimality gap of 3.3 × 10−9, compared with the width-N single-pole ceiling −1.3176.The learned spectrum contains one pole per task rate and one zero-eigenvalue feedthrough channel per mode.Source: D. Optimality Gap and Feedthrough… · pp. 17–18
  • For one task mode, width two closes the single-ray gap to 3.29 × 10−9 in all twenty runs, while extra width does not improve recruitment.Width one remains at a gap of 2.85 × 10−1 in twenty of twenty runs.Source: D. Optimality Gap and Feedthrough… · p. 18

General covariance families

  • Beyond AR(1), covariance structure determines the correction around the task-rate geometric tail, and finite-order families require feedthrough, poles, or seasonal components accordingly.The selected components close the single-ray gap below 10−6 across reported finite-order families.Source: D. Optimality Gap and Feedthrough… · p. 18

Feedthrough capacity

  • A single recurrent mode cannot represent the correlated optimum, whereas one eigenvalue-zero feedthrough dimension per task mode can repair the gap.The feedthrough passes the current input directly to the output while adding no recurrent memory.Source: D. Optimality Gap and Feedthrough… · pp. 16, 18

E. Real-Data Validation

The paper tests its threshold and representation claims on real and synthetic covariance structures. Across nineteen series, learned crossings match the measured newest-step thresholds within the stated bisection bias, while scope-condition cases exhibit first-order transitions instead.

Protocol

  • The real-data protocol standardizes or differences series, uses overlapping windows of length T = 20, and estimates temporal covariance from each window matrix.Series are sampled at their natural resolutions across finance, weather, hydrology, air quality, macroeconomics, sunspots, CO2, Niño 3.4, and EEG.Source: E. Real-Data Validation · p. 18

Threshold validation

  • Across nineteen series, thresholds span 0.1438 to 0.9828, and every learned crossing lies within 1.5 × 10−4 of its measured threshold.The measured threshold is the correlation of the two newest covariance steps.Source: E. Real-Data Validation · p. 19, Table S2 · p. 20
  • The newest-step threshold, rather than average lag-one correlation, predicts the crossing: US inflation has lag-one value 0.5941 but threshold 0.6152.Its crossing follows the newest-step threshold under the nonstationary formulation.Source: E. Real-Data Validation · p. 19

Invariance tests

  • Synthetic covariances sharing threshold 0.6 show crossing movement of at most 6.0 × 10−5 when sequence length and longer-range covariance tails vary.This supports the predicted invariance to window length and correlations beyond the newest pair.Source: E. Real-Data Validation · p. 20

Negative thresholds

  • Negatively correlated newest samples produce negative thresholds, so no memory transition occurs within λ ∈ (0, 1); both corresponding series already show positive settled dials at λ = 0.01.The measured thresholds are −0.154 for daily S&P returns and −0.041 for daily Bitcoin returns.Source: E. Real-Data Validation · p. 20

Scope condition

  • Strong periodicity or extreme persistence can produce first-order transitions in which a distant energy minimum overtakes the endpoint before the local memory-release condition fires.This is a scope condition on the complete, continuous transition described for AR(1)-like covariances.Source: E. Real-Data Validation · p. 20

Related papers

Loading 2609.00420v1…