Source-linked AI summary
Learning reveals invisible structure in low-rank RNNs
Yoav Ger, Omri Barak
TL;DR
Low-rank overlap models describe network function, but it has remained unclear whether they can also provide a low-dimensional account of learning. This paper derives overlap-space learning dynamics and distinguishes function-determining overlaps from functionally silent ones. It shows how learning can reveal connectivity differences between functionally equivalent networks and how silent overlaps can encode training history.
Problem
It remains unclear whether the low-rank overlap view that describes network function can also describe learning while retaining a low-dimensional representation.
Source: §1 · p. 2Method
The paper derives closed-form gradient-descent ODEs in overlap space and separates loss-visible overlaps from loss-invisible overlaps.
Source: §1 · p. 2Results
Learning can resolve connectivity ambiguity between functionally equivalent networks, while loss-invisible overlaps can encode training history; memory is generally unreliable in linear networks but emerges more readily in nonlinear ones.
Source: §1 · p. 2Takeaways & Limitations
Learning trajectories may serve as non-invasive probes of circuit structure, and loss-invisible directions are candidate memory variables for training history.
Source: §7 · p. 9
Abstract
from arXiv · showhide
Learning in neural systems arises from synaptic changes that reshape the representations underlying behavior. While low-rank recurrent neural networks (RNNs) have emerged as a powerful framework for linking connectivity to function, a theoretical understanding of their learning process remains elusive. Here, we extend the low-rank framework from activity to learning by deriving gradient-descent dynamics directly in a reduced overlap space. We formulate a closed-form, low-dimensional system of ODEs that governs learning in this space, exact for linear RNNs and asymptotically exact for nonlinear RNNs in the large-$N$ Gaussian limit. Central to our analysis is a distinction between two classes of overlaps: loss-visible overlaps, which fully determine network activity, output, and loss, and loss-invisible overlaps, which do not affect function but are required to describe learning. We illustrate the consequences of this decomposition through two phenomena. First, we show that learning can serve as a perturbation that exposes differences in connectivity between functionally equivalent networks. Second, we show that loss-invisible overlaps can act as memory variables that encode training history, and characterize the conditions under which this occurs. Finally, we present several testable predictions for biological learning experiments derived from our theory.
1 Introduction
The introduction frames learning as a challenge of linking high-dimensional synaptic changes to lower-dimensional behavior, and asks whether low-rank overlap descriptions can capture learning as well as function. It presents closed-form overlap dynamics and a distinction between function-determining and functionally silent overlaps, with consequences for probing connectivity and retaining training history.
1 Introduction
- High-dimensional synaptic changes and lower-dimensional behavior make the mapping from function to connectivity ill-posed, raising questions about degeneracy and identifiability.The mismatch between these scales is one theoretical reason that linking circuit-level plasticity to behavior is challenging.Source: §1 · p. 1
- Whether the low-rank overlap view of network function can also describe learning in a low-dimensional way remains unclear.Prior learning-dynamics approaches have largely been developed outside the low-rank framework.Source: §1 · pp. 1–2
- The paper derives closed-form overlap-learning ODEs that are exact for linear RNNs and asymptotically exact for Gaussian nonlinear RNNs as N grows.The authors describe this as the first analytical description of learning in nonlinear task-trained RNNs, to their knowledge.Source: §1 · p. 2
- Loss-visible overlaps determine activity, output, and loss, while loss-invisible overlaps are functionally silent but shape learning, with the boundary depending on activation.The distinction is between overlaps sufficient to determine current function and additional overlaps that constrain learning dynamics.Source: §1 · p. 2
- Learning can resolve connectivity ambiguity between networks with identical function and expose structural differences, while loss-invisible overlaps can encode training history.Memory is generally unreliable in linear networks and depends on the learning rule, whereas it emerges more readily in nonlinear networks.Source: §1 · p. 2
2 Preliminaries
The preliminaries set up a rate-based RNN trained by gradient descent and motivate reducing its high-dimensional parameter dynamics to variables that determine behavior and loss. For low-rank networks, the same input-output function can be represented through a small set of scalar overlaps.
2 Preliminaries
- The model is a rate-based RNN with hidden state h, element-wise activation ϕ, recurrent connectivity W, scalar input through m, and linear readout through z.The setup focuses on a single input and output, with multiple inputs or outputs described as straightforward extensions.Source: §2 · p. 2
- The trainable parameters θ = {m, W, z} follow gradient flow, ˙θ = −∇θL, for squared-error loss against a target.The gradient-flow limit is η → 0.Source: §2 · pp. 2–3
- A low-rank RNN can capture the same input-output function as a high-dimensional RNN using a small set of scalar overlaps.Figure 1 contrasts the parameter-space network with its effective overlap description.Source: Figure 1 · p. 3
3 Low-rank linear RNN
For rank-1 linear RNNs, the network reduces exactly to a two-dimensional activity system governed by four loss-visible overlaps. Learning nevertheless depends on additional overlaps through a parameter-induced metric, so optimizing the loss directly in visible-overlap space does not reproduce parameter-space learning.
3 Low-rank linear RNN
- The N-dimensional rank-1 linear RNN reduces exactly to a two-dimensional activity system determined by four loss-visible overlaps.Those overlaps are σ = (σ_zm, σ_zu, σ_vm, σ_vu), which fully determine within-episode dynamics and loss.Source: §3 · pp. 3–4
- Parameter-space gradient flow induces overlap dynamics preconditioned by a positive-semidefinite Gram matrix G(θ), rather than naive gradient descent on overlaps.G captures the geometry inherited from parameter space and can be computed in closed form for rank-1 RNNs.Source: §3 · p. 4
- The learning metric depends on six additional overlaps and parameter norms that do not contribute to the loss.These include σ_mu and σ_zv and the squared norms of the four parameter vectors; they are grouped as loss-invisible overlaps.Source: §3 · p. 4
- Visible and invisible overlaps together provide a closed 10-dimensional description of learning, with invisible-overlap dynamics derived explicitly.The full set of ten quadratic overlaps jointly governs both classes of overlap dynamics.Source: §3 · p. 4
- Full parameter-space gradient descent and the derived 10D overlap ODE match exactly on the filter task, while direct optimization of L(σ) yields qualitatively different loss dynamics.The task trains a rank-1 linear RNN to emulate a first-order exponential filter driven by white noise.Source: §3 · p. 4
4 Implications of visible and invisible overlaps
Loss-visible overlaps determine function, while loss-invisible overlaps leave function unchanged but shape learning. This distinction explains how learning can expose connectivity differences and when invisible overlaps do—or do not—retain training history.
4.1 Learning reveals hidden degeneracy
- Networks with identical loss-visible overlaps can produce indistinguishable activity and outputs yet diverge during learning, exposing differences in their loss-invisible connectivity.For the illustrated pair, one network also develops a transient loss plateau absent in the other; the reduced theory captures both learning dynamics.Source: §4.1 · pp. 4–5, Figure 2 · p. 5
4.2 Memory and its absence in invisible overlaps
- Under vanilla gradient descent, retraining on task A restores both visible and invisible overlaps, so the linear-network protocol does not retain training history in the invisible set.The conserved matrix K constrains trajectories to initialization-dependent invariant manifolds; returning to the original visible solution therefore restores the invisible overlaps.Source: §4.2 · pp. 5–6, Figure 3 · p. 6
- Adding label noise breaks conservation of K and drives invisible overlaps away from their original values while leaving visible overlaps unchanged.This creates a directed drift in the invisible subspace rather than a change in the task-visible solution.Source: §4.2 · p. 6, Figure 3 · p. 6
5 Low-rank nonlinear RNN
In nonlinear low-rank RNNs, the visible–invisible boundary changes, and the overlap dynamics can be tested against high-dimensional learning. The theory closely tracks flip-flop training under a Gaussian regime, while noisy overlap measurements identify training history through the invisible set.
5 Low-rank nonlinear RNN
- In the nonlinear Gaussian-limit analysis, σ_mu, ||m||2, and ||u||2 become loss-visible, changing the linear case’s visible–invisible separation.These overlaps enter the dynamics through Δ(t), so the loss no longer has the disjoint bilinear parameter dependence described for the linear case.Source: §5 · pp. 6–7
5.1 Flip-flop task
- On the flip-flop task, the 10D overlap theory closely matches high-dimensional RNN loss dynamics, but its validity depends on approximately Gaussian weight components.The Gaussian structure holds numerically for sufficiently small learning rates; larger rates or Adam can produce deviations and theory–simulation discrepancies.Source: §5.1 · p. 7, Figure 4 · p. 7
5.2 Memory is encoded in the invisible overlaps
- With small noise added to overlap measurements, only loss-invisible overlaps robustly decode whether training began with task A or B; loss-visible decoding falls to chance.Without noise, both sets appear informative because of infinitesimal differences from imperfect convergence.Source: Figure 4 · p. 7, §5.2 · p. 8
6 Related work
Prior low-rank RNN work studied designed computations and learning-induced structure, while analytical RNN-learning studies used task or initialization simplifications. This work relates its overlap-space approach to function-space and natural-gradient perspectives, and to invariants that constrain memory.
Learning dynamics of RNNs
- Earlier analytical RNN-learning studies used assumptions such as timescale separation, frozen parameters, or special initialization, whereas this analysis lets all parameters evolve under architectural constraints.It also extends the analysis to nonlinear task-trained RNN learning dynamics.Source: §6 · p. 8
Parameter and function space duality
- The overlap-space approach tracks quantities beyond the output, and its matrix G maps parameter-space gradients to overlap dynamics with a geometry resembling natural-gradient preconditioning.Unlike a prescribed distance metric, G arises from the low-rank parameterization.Source: §6 · p. 9
Invariants, symmetries, and drift
- In linear networks under gradient flow, conserved quantities constrain loss-invisible directions and prevent them from storing training history; breaking those symmetries can enable memory.The discussion also relates label-noise drift to biases toward flatter, lower-norm, or balanced regions of a solution manifold [32] [52].Source: §6 · p. 9
7 Discussion
The discussion frames low-rank overlap dynamics as a tractable link between connectivity changes and network function. It highlights learning trajectories as structural probes and loss-invisible overlaps as candidate records of training history.
7 Discussion
- Learning trajectories can reveal structural differences between functionally equivalent networks, while loss-invisible directions may encode training history and future learning capacity.The authors propose these as testable predictions for biological learning experiments.Source: §7 · p. 9
- The analysis focuses on rank-1, purely low-rank connectivity; higher rank is possible but increases overlap counts quadratically and complicates the dynamics.The nonlinear analysis also assumes parameter-vector components remain approximately Gaussian during training, an assumption not strictly preserved.Source: §7 · p. 9
- The framework offers a tractable link between connectivity changes and network-function evolution, with relevance to degeneracy, memory, and drift.The authors frame these implications for neuroscience and machine-learning theory.Source: §7 · p. 9
A.2 Across-episode learning dynamics
- In the linear rank-1 case, six additional loss-invisible overlaps are needed to close the learning dynamics, yielding a closed 10-dimensional ODE system.The equations describe the learning trajectory in overlap space.Source: §A.2 · p. 16
A.3 Learning dynamics in matrix form
- The closed overlap dynamics can be written as a matrix-form update induced by the parameter-to-overlap Jacobian and its Gram matrix.The visible-overlap update is σ̇ = −G(θ)∇σL, with G(θ) = D(θ)D(θ)⊤.Source: §A.3 · p. 16, §A.2 · p. 16
A.4 Filter task
The filter-task analyses characterize exact target-filter solutions and show how overlap-space learning, invariants, and nonlinear dynamics relate to high-dimensional RNN training. The later sections extend the analysis to a nonlinear flip-flop task and its reduced learning equations.
A.4.1 Training details and RNN initialization
- Numerical simulations trained an Euler-discretized continuous-time rank-1 RNN (N=500) by gradient descent (η=5 × 10−3) to match a first-order filter (a⋆=1, c⋆=0.2), using impulse-response input over T=20 s.Source: §A.4.1 · p. 17
A.4 Filter task
- The filter-task loss and overlap trajectories from high-dimensional simulation agree with the corresponding overlap-space ODE predictions.The figure also compares simulated and predicted final overlap values.Source: Figure 5 · p. 17
A.4.2 Network impulse response and exact solution
- The target one-pole filter is matched by eliminating the unwanted mode and setting the remaining pole and amplitude to c⋆ and a⋆.The solution constraints leave a one-dimensional manifold of equivalent visible-overlap minima because the function depends on σzu and σvm through their product.Source: §A.4.2 · p. 18
A.4.3 Exact gradient calculation
- For the filter task, loss gradients with respect to the loss-visible overlaps can be derived in closed form.Source: §A.4.3 · p. 18
A.5 An invariant under gradient flow
- Under gradient flow, conserved quantities and task-specific constraints tie loss-invisible overlap evolution to initialization, yielding exact A–B–A recovery.With noisy training, overlaps drift most strongly in the loss-invisible set; Adam also violates the invariant and permits new invisible-overlap values.Source: §A.5 · p. 19, Figure 6 · p. 20, Figure 7 · p. 20
B.1 Within-episode dynamics
- For nonlinear rank-1 RNNs, the within-episode N-dimensional system reduces to a smaller system, and σmu and the norms of m and u become loss-visible.These quantities affect network dynamics and output through the variance ∆(t).Source: §B.1 · p. 21
B.2 Across-episode learning dynamics
- Across-episode learning derives parameter gradients by the chain rule and updates both loss-visible and loss-invisible overlaps.The equations are also written in compact matrix form.Source: §B.2 · p. 22, §B.3 · p. 23
B.4 Flip-flop task
- The nonlinear flip-flop task requires stable output that changes only after brief signed input pulses.The task is designed to require bistability, which linear systems lack.Source: §B.4 · p. 23
B.4.1 Training details, RNN initialization and Gaussian assumption
The nonlinear flip-flop validation relies on approximately Gaussian parameter vectors during training. Small-learning-rate gradient flow retains this property, whereas larger-step gradient descent and Adam do not, and the effective theory then fails to match simulated output.
B.4.1 Training details, RNN initialization and Gaussian assumption
- The flip-flop simulations use a rank-1 RNN with N=1000, learning rate η=0.05, and masked MSE that ignores pulses and short transients.The mask focuses learning on maintaining stable fixed-point outputs.Source: §B.4.1 · p. 23
- Under small-learning-rate gradient flow, parameter components remain approximately Gaussian and the effective model captures simulated outputs; under gradient descent and Adam, Gaussianity breaks down and the theory fails to capture output.Source: §B.4.1 · p. 23, Figure 8 · p. 24
B.5 History-dependent training protocol
The history-dependent protocol trains twin networks on different initial tasks and then on a shared teacher task. After the shared training, the supplied figure shows convergent loss-visible overlaps but distinct loss-invisible overlaps.
B.5 History-dependent training protocol
- The protocol uses flip-flop Task A, stimulus-integration Task B, and a teacher-student Task C, with 30,000 epochs in each of two phases.Phase A/B uses an initially η=0.01 learning rate, reduced when loss falls below 0.015; Phase C uses η=0.001.Source: §B.5 · p. 24
- Task B integrates noisy coherence input and targets a continuous response proportional to stimulus strength during the response period.The coherence levels are ±2, ±8, and ±16.Source: §B.5 · p. 24
- After shared Task C training, the two networks’ loss-visible overlaps converge to the same values while their loss-invisible overlaps remain distinct.The figure follows networks initially trained on Task A or B before retraining on C.Source: Figure 9 · p. 25, Figure 10 · p. 26
C Derivation of the augmented Gram matrix ¯G
The augmented Gram matrix organizes overlap learning across all quadratic scalars formed from the four connectivity vectors. Its linear and nonlinear structures differ in whether visible and invisible overlaps remain separated or become coupled.
C Derivation of the augmented Gram matrix ¯G
- The augmented overlap set comprises six pairwise overlaps and four squared norms, and its 10 × 10 Gram matrix is defined from the Jacobian of these scalars.Source: C Derivation of the augmented Gram… · p. 27
- The augmented Gram matrix applies to RNNs formed from the four vectors, while nonlinearity changes the loss and the boundary between visible and invisible overlaps.In the linear case, visible-overlap updates depend on invisible quantities and vice versa; in the nonlinear case, mixed visible–invisible coupling appears.Source: C Derivation of the augmented Gram… · p. 27, Figure 11 · p. 28
D Rank-2 linear RNN
This rank-2 extension expresses learning in a closed overlap system and validates its predictions against high-dimensional training.
D Rank-2 linear RNN
- With zero initial conditions, the rank-2 network's dynamics remain in the three-dimensional subspace spanned by m, u1, and u2.The parameter set comprises six vectors: m, z, u1, v1, u2, and v2.Source: D Rank-2 linear RNN · p. 29
- Nine loss-visible overlaps and the loss-invisible overlap dynamics form a closed 21-dimensional system that fully characterizes rank-2 learning.Source: D Rank-2 linear RNN · pp. 29–30
- For the damped-sinusoidal target, full-RNN loss and overlap trajectories align perfectly with the scalar ODE predictions.The target frequency is 2 and its damping rate is 0.3.Source: Figure 12 · p. 30, D Rank-2 linear RNN · p. 30
- For rank r with multiple inputs and outputs, the total overlap count scales as O((2r + min + zout)^2) yet remains independent of network size N.The resulting dynamics are described as tractable despite quadratic growth with rank and input/output channel counts.Source: D Rank-2 linear RNN · p. 30