Learning infinite context windows in recurrent architectures via spatial neural computing

1 Introduction

• RNNs provide O(1) memory and O(T) computation, but vanishing or exploding gradients constrain their practical ability to capture long-range dependencies.Gated architectures mitigate gradient decay but add complexity and computational overhead.Source: §1 · pp. 1–2
• Transformers model long-range dependencies with self-attention, but require O(T) key-value memory and O(T^2) attention computation.The cited introduction also notes that dissipative dynamics in many state-space and oscillator approaches do not maintain nonvanishing gradients over arbitrarily long horizons.Source: §1 · p. 2
• The paper asks how recurrent architectures can maintain nonvanishing gradient modes despite input-dependent nonlinear perturbations.Its analysis identifies a second-order structure in which input-independent coefficients can fix gradient-mode magnitudes while inputs perturb other dynamics.Source: §1 · p. 2
• SpatialRNN mediates neuron communication through a PDE-governed spatial medium, making the model equivalent to an infinite-history RNN with a parameter-efficient, unbounded receptive field.Constructive conditions control gradient modes on circles of prescribed radii, supporting retention at unit radius or forgetting at smaller radii.Source: §1 · p. 2

2 Spatial Computing Model

• SpatialRNN replaces direct hidden-state communication with a spatially evolving medium, where local neuron interactions drive structured information propagation over time.The medium follows a second-order PDE and is discretized into a second-order linear difference equation; neuron activity acts as its forcing term.Source: §2 · p. 3

2.1 Higher-Order and Standard RNNs

• A finite-order HORNN explicitly connects the current state to d past states, but increasing d adds parameters and leaves dependencies beyond d to propagate through intermediate states.For d = 1, the model reduces to a standard RNN.Source: §2.1 · pp. 3–4, Figure 1 · p. 4

2.2 Equivalence between SpatialRNNs and ∞-order RNNs

• The spatial medium is equivalent to a convolution over the entire hidden-state history, implicitly generating lag-specific matrices instead of storing each one separately.The model uses four fixed-size matrices for this implicit parameterization and compresses history into ψ_t and ψ_{t−1}, restoring O(1) inference memory.Source: §2.2 · pp. 4–5
• An unbounded receptive field means any past state can in principle influence the current state, but does not guarantee nonvanishing influence or lossless retention of arbitrary history.Long-lived influence depends on gradient dynamics, and a fixed-dimensional finite-precision state cannot losslessly preserve an indefinitely long history.Source: §2.2 · p. 5

3 Gradient Dynamics and Conditions for Long-Term Memory

• Marginal stability of every frozen-time system is necessary, but not sufficient, for marginal stability of the full LTV system under arbitrary perturbation switching.The frozen-time approximation holds the instantaneous transition matrix fixed at a reference time.Source: §3 · p. 6

3.1 Nonrobustness of RNN and HORNN Gradients

• Figure 2 contrasts input-sensitive standard-RNN spectra with SpatialRNN spectra whose eigenvalue magnitudes can remain invariant, including on the unit circle.The SpatialRNN comparison applies when Theorem 3.3's conditions are satisfied.Source: Figure 2 · p. 7, §3.1 · p. 7
• Finite-order models cannot robustly retain unit-modulus modes when activation derivatives approach zero; this parameterization-independent result extends across orthogonal, nonnormal, and diagonal RNNs.For any finite order, the proposition states there is a perturbation for which no gradient eigenvalue lies on the unit circle.Source: §3.1 · p. 7

3.2 Robustness of SpatialRNN Gradients

• Under Theorem 3.3's block constraints, SpatialRNN gradient eigenvalues have modulus √α_i for every admissible nonlinear perturbation.The theorem specifies structural constraints on the medium update matrices and forcing terms.Source: §3.2 · p. 8
• Setting α_i = 1 yields marginal stability across all admissible nonlinear perturbations, while α_i < 1 produces exponential gradient decay at a controlled rate.The α_i values may be fixed in advance or learned via BPTT.Source: §3.2 · p. 8
• Unlike finite-order counterparts, infinite-order RNNs can achieve robust marginal stability when their delay matrices are appropriately constrained.Source: §3.2 · p. 8

Copy task

• ≈99%: SpatialRNN maintains near-perfect validation accuracy across copy-task sequence lengths, while finite-order HORNN accuracy eventually degrades.Larger HORNN receptive fields delay but do not prevent deterioration; the comparison includes models with a second-order HORNN matching SpatialRNN's order and state dimension.Source: §4 · p. 9

Permuted sequential MNIST (psMNIST)

• On psMNIST, SpatialRNNs achieve 96.3 ± 0.1% accuracy, the highest among single-layer recurrent baselines, and attain this performance with roughly four to ten times fewer parameters.Source: §4 · p. 10, Table 1 · p. 10

Noise padded CIFAR-10 (npCIFAR10)

• SpatialRNN performance is invariant to appended-noise amount on npCIFAR10, with accuracy attributed mainly to initial-sequence spatial feature extraction rather than temporal degradation.Figure 4 reports validation accuracy for varying T.Source: §4 · p. 10, Figure 4 · p. 11

Conclusion

• The spatial medium implicitly encodes the full history of hidden states, giving an unbounded receptive field with a fixed number of parameters.Under suitable constraints, gradient modes can remain on circles of prescribed radii despite input-dependent nonlinear perturbations.Source: §5 · p. 11

Theoretical perspective

• Robust marginal stability depends on how nonlinearities enter gradient dynamics: SpatialRNN separates input-dependent perturbations from coefficients that set gradient-mode magnitudes.For finite-order RNNs, nonlinear terms directly perturb coefficients governing modal decay.Source: §5 · p. 11

Related work

• Unlike local oscillator-based wave models, SpatialRNN encodes traveling waves in a shared medium; the approaches coincide only under restrictive conditions.The related work notes that stability in those wave models depends directly on discretization parameters.Source: §5 · p. 11

Limitation

• Frozen-time stability guarantees are necessary but insufficient for marginal stability under arbitrary switching, so gradients may still grow over long horizons.Additional structural constraints can enforce asymptotic stability, while necessary and sufficient conditions for LTV marginal stability remain open.Source: §5 · pp. 11–12

Limitations

• Memory capacity remains bounded by medium dimension and numerical precision, while each sequential SpatialRNN step uses more matrix-vector products than a standard RNN.The nonlinear recurrence is evaluated sequentially, unlike associative scans in linear SSMs.Source: §5 · p. 12

Future work

• Future work proposes lifting SSM recurrences to second order and targeting streaming tasks requiring fixed-size state, long retention, and selective forgetting.Suggested settings include sensor monitoring, long-horizon control, and spatiotemporal signal prediction.Source: §5 · p. 12

Broader Impact Statement

• SpatialRNN may reduce long-horizon inference memory costs by retaining dependencies in a fixed-size state; practical energy savings remain to be established.The authors also expect relevance to analog and wave-based physical computing substrates.Source: Broader Impact Statement · p. 12

A.1 Proofs of Main Theoretical Results

• Matching coefficients in the history-dependent recurrence gives W1 = B, W2 = WW1, and Wk = WWk−1 + WWk−2 for k ≥ 3.These relations construct the infinite sequence of delay matrices from the SpatialRNN parameters.Source: §A.1 · p. 16
• The first-order system cannot be robustly marginally stable across the allowed activation derivatives: a choice of parameters yields no characteristic roots on the unit circle.The proof bounds the spectral radius of a polynomial matrix on the unit circle, then chooses a scalar larger than that bound to contradict robust marginal stability.Source: §A.1 · pp. 16–17
• Block-triangular second-order dynamics reduce to independent subsystems, whose characteristic roots are governed by quadratic equations.Under the stated block and symmetry assumptions, the full system’s eigenvalues are the union of the subsystem eigenvalues; the derived roots have modulus √α_i when the discriminant condition holds.Source: §A.1 · p. 17

A.2 Derivation of the SpatialRNN Gradient Dynamics

• The SpatialRNN gradient equations follow by applying the chain rule to the hidden-state and medium-state updates, then substituting one recurrence into the other.The resulting coupled linear time-varying recurrence is written in matrix form as Eq. (11).Source: §A.2 · p. 18

• The infinite-order RNN’s gradient dynamics are equivalent to the SpatialRNN’s because the two models are equivalent when their delay matrices follow the stated recurrence.The result uses invertibility of Wψ and the delay-matrix relation from Eq. (6).Source: §A.3 · p. 18
• The appendix tests eigenvalues of the infinite-order recurrence through a polynomial matrix formed from its delay transition matrices.It factors out Wψ under the stated invertibility assumption and relates the recurrence’s characteristic equation to the poles of its gradient dynamics.Source: §A.3 · p. 18

A.4 Emergence of Marginal Stability During SpatialRNN Training

• Learning remains at chance until α4 approaches 1, when training accuracy rises sharply on the copy task with sequence length T = 128.The authors report that training can be slow from random initialization and fix α_i = 1 for at least one block in experimental evaluations.Source: §A.4 · p. 19

A.5 Connection between SpatialRNN and Oscillator Networks

• For block-triangular weights, the gradient eigenvalue problem decomposes into independent blocks, allowing robust control of each block’s eigenvalue moduli while modes can vary over time.The varying eigenvectors promote information mixing and higher model expressivity.Source: §A.5 · p. 19
• Simultaneous diagonalization keeps eigenvectors fixed over time, preventing information mixing between modes and limiting expressivity; the authors therefore do not use this approach experimentally.This is presented as an alternative to the block-triangular construction.Source: §A.5 · p. 19
• Under the oscillator-mode conditions, each mode has two simple complex-conjugate eigenvalues with time-invariant modulus √α_i, for any allowed time-varying sequence matrix.The conditions bound |μ_i| + Φ′|β_iγ_i| below 2√α_i, ensuring a negative discriminant.Source: §A.5 · pp. 19–20

• Frozen-time marginal stability is necessary but not sufficient for the switching LTV gradient equations, and gradient norms show mild long-horizon drift.The appendix illustrates the drift with the gradient-norm plot in Figure 6.Source: §A.6 · p. 20, Figure 6 · p. 21
• The appendix’s sufficient Lyapunov conditions impose bounds on the matrix norms to ensure uniform stability under arbitrary time variation.The bounds are applied to the perturbation matrices across time and yield a global time-invariant Lyapunov function.Source: §A.6 · p. 22
• Under the stated block and matrix assumptions, the appendix constructs blockwise Lyapunov functions whose sum establishes uniform asymptotic stability for the full LTV system.A fixed positive-definite matrix for each block must make its energy difference strictly negative at every step.Source: §A.6 · pp. 20–22
• Small but nonvanishing state-to-medium and medium-to-state matrix norms prevent exploding gradients at the cost of some gradient decay, so the resulting LTV dynamics are asymptotically but not necessarily marginally stable.The stronger norm constraint needed for marginal stability would make the SpatialRNN untrainable, according to the appendix.Source: §A.6 · pp. 20, 22
• For α_i = 1 across blocks, the gradient transition product has norm at least 1, and its volume preservation prevents medium gradients from vanishing in every direction.Volume preservation is necessary but not sufficient for marginal stability because growth in some directions can be offset by decay in others.Source: §A.6 · p. 22

A.7 Empirical LTV Gradient Dynamics along Input Trajectories

• The analysis propagates state-to-state Jacobians along real input trajectories using activation derivatives realized by each input sequence.Unlike loss gradients, this measurement does not depend on the loss, readout, or label, so it directly probes the time-varying dynamics.Source: §A.7 · p. 23
• For each sequence, a log-linear fit to Jacobian Frobenius norms for m ≥50 estimates the top Lyapunov exponent of the time-varying gradient dynamics.The trained model and its initialization were evaluated on 100 psMNIST validation sequences in double precision, with recursion rescaled at each step to avoid overflow or underflow.Source: §A.7 · p. 23
• A slope of 7.98·10^-5 per step in the trained model corresponds to only ≈6% Jacobian-norm change across 783 steps, versus ≈23% at initialization.Training moves the model closer to the marginal regime predicted by the frozen-time analysis and reduces residual drift from time-varying dynamics.Source: §A.7 · p. 23, Figure 7 · p. 23

B Symplectic Form Preservation of the SpatialRNN

• Symplectic dynamics preserve phase-space volume and are invertible, unlike dissipative dynamics that contract volume and irreversibly lose information.The appendix connects this property to numerically stable, time-reversible evolution and recovery of past states.Source: B Symplectic Form Preservation of the… · pp. 23–24
• The symplectic condition for the one-step map is M^TJM = J in canonical position-momentum coordinates, with Proposition B.1 specifying conditions on the update matrices.The wave-equation update satisfies the proposition's conditions and preserves the symplectic form.Source: B Symplectic Form Preservation of the… · p. 24

B.1 Exact Backward Reconstruction of the SpatialRNN States

• Under Theorem 3.3 conditions, the full state history can be reconstructed from the final boundary states by iteratively recovering past states.The second-order medium evolution is reversible; in exact arithmetic, past states remain recoverable even when α_i < 1.Source: §B.1 · p. 25

B.2 Numerical Precision of the Reconstruction

• In finite precision, backward reconstruction amplifies round-off by α_i^-1/2 per step, so the useful horizon depends on the tolerance and machine epsilon.This bound follows from inverting medium modes of modulus √α_i.Source: §B.2 · p. 25
• O(1)-memory BPTT is practical only when every block is non-dissipative; blocks with α_i < 1 require caching or checkpointing every K < m_max steps.Checkpointing yields memory use O(T/K).Source: §B.2 · p. 26
• Below 10^-5 in single and 10^-13 in double precision across the horizon, non-dissipative reconstruction stays accurate; dissipative blocks grow 1.42 per step, exceeding 10^-3 after 44 and 101 steps, respectively.The measured amplification matches the predicted 1/0.5 = 1.414 rate; Figure 8 compares errors over backward-recursion depth.Source: §B.2 · p. 25, Figure 8 · p. 26

C.1 SpatialRNN implementation details

• SpatialRNN uses tanh, fixes α_1 = 1 while learning other decay parameters, and applies structural masks and spectral-norm projections to enforce stability constraints.The implementation initializes projection matrices with Xavier uniform weights, medium block diagonals orthogonally, and B and Σ as identity matrices.Source: §C.1 · pp. 26–27

C.2 HORNN implementation details

• Sparse-delay HORNNs use selected recurrence delays, reducing dense-delay parameter overhead while limiting the effective temporal receptive field to d_max = max(T).The appendix specifies tanh activations and Xavier initialization for the input, output, and sparse recurrence matrices.Source: §C.2 · p. 27

C.3 Copy Task

• The copy task tests retention of five randomly sampled data tokens across a zero-padded distractor sequence before five trigger tokens prompt retrieval.Models are trained with cross-entropy over the final five retrieval steps, using 3,000 training sequences per epoch and early stopping.Source: §C.3 · p. 27

C.4 Permuted Sequential MNIST (psMNIST)

• psMNIST flattens MNIST images into 784-step sequences and applies a fixed random permutation to isolate reliance on long-range temporal dependencies.The model produces one of 10 classes after processing the sequence; training uses three independent initializations.Source: §C.4 · pp. 27–28

C.5 Noise Padded CIFAR-10 (npCIFAR10)

• npCIFAR10 represents each image as 32 row-wise steps, then appends standard-normal noise vectors to reach the evaluated sequence length.At the primary T = 1000 setting, the model classifies after the full sequence; the described length ablation tests whether distractor quantity affects performance.Source: §C.5 · pp. 28–29

C.6 Hardware and Compute

• The reported experiments ran on CPU resources only, with approximately two weeks of total compute estimated on the workstation.The copy-task experiments used approximately 1,600 CPU-thread-hours, psMNIST approximately 1,600, and npCIFAR10 approximately 3,200.Source: §C.6 · pp. 29–30