PaperlayerView exact revision on arXiv

Source-linked AI summary

Prospective Coding Improves Learning in Deep Continuous-Time Recurrent Networks

Shivang Rawat, Mirko Morello, Flaviano Morone, David J. Heeger

arXiv:2609.04134v1cs.LGcs.NEq-bio.NC

TL;DR

Deep continuous-time recurrent networks gain memory from temporal integration but suffer delayed bottom-up signals and depth-dependent learning attenuation. The paper develops RQFs and a parameter-free prospective-input correction for diagonal SSMs, finding that prospective variants match or outperform controls across tested models and configurations while enabling strong long-sequence results. The authors conclude that RQFs are parameter-efficient recurrent substrates and prospective coding is an input-side correction for deep continuous-time recurrent networks.

  • Problem

    Deep temporal integration provides memory but delays bottom-up signals and attenuates top-down errors in stacked continuous-time recurrent networks.

  • Method

    The paper develops biologically motivated complex-valued RQF filters as diagonal SSMs and adds a parameter-free two-tap prospective-input correction that preserves recurrent transitions and parallel scans.

  • Results

    Prospective variants match or outperform non-prospective controls in every tested configuration, including a 96.09% Speech Commands RQF with 31.9k parameters.

  • Takeaways & Limitations

    RQFs provide a parameter-efficient recurrent substrate, while prospective-input coding offers an input-side correction for deep continuous-time recurrent networks.

  • Takeaways & Limitations

    The correction is not directly applicable to discrete-time gated RNNs, and its empirical gain can vary with nonlinearities, residual connections, normalization, input-dependent parameters, and look-ahead scale.

Abstract

from arXiv · show

Temporal integration gives continuous-time recurrent networks memory, but in deep stacks it also delays bottom-up signals and attenuates top-down errors. We develop Recursive Quadrature Filters (RQFs), biologically motivated complex-valued temporal filters that are a special case of diagonal state-space models (SSMs), and ask whether this failure mode can be addressed by making each layer's bottom-up input prospective. Starting from an energy model, we derive the RQF dynamics and show that each RQF is a band-pass filter whose learnable parameters control its tuning frequency and bandwidth. We then make each layer's bottom-up input prospective using a parameter-free two-tap update that leaves the recurrent transition and parallel scan unchanged. We extend this correction to general diagonal SSMs and show that it mitigates depth-dependent gradient attenuation when temporal gradients are truncated, i.e., spatial-only backpropagation. We evaluate the intervention in RQFs, S5, and ORGaNICs (a nonlinear gated RNN) trained using full backpropagation through time (BPTT) and spatial-only backpropagation. Under full BPTT, prospective variants match or outperform their non-prospective controls in every model and configuration. A non-residual width-32 six-layer RQF reaches 96.09% accuracy on raw-audio Speech Commands with 31.9k parameters; a width-64 six-layer RQF reaches 83.56% on the 16,384-step Path-X task. These results identify RQFs as a parameter-efficient recurrent substrate and prospective-input coding as an input-side correction for deep continuous-time recurrent networks.

1 Introduction

Deep continuous-time recurrent stacks trade temporal memory for delayed bottom-up signals and attenuated top-down learning signals. The paper introduces RQFs and prospective-input coding to address this trade-off while preserving efficient diagonal-SSM computation.

  • Motivation: Deep leaky-integrator stacks delay bottom-up signals and make top-down teaching signals increasingly out of date across layers.The effective relaxation time grows roughly as Lτ, creating a learning problem for fast-varying signals.
  • RQF architecture: RQFs are biologically motivated complex-valued band-pass filters with learnable phase-prior frequency and bandwidth.They are derived from an energy-based cortical theory and extend feature extraction over time.
  • RQF architecture: RQFs are constrained diagonal SSMs, inheriting parallel-scan and FFT-convolutional training algorithms.This makes long-sequence tasks such as 16,384-step Path-X computationally accessible.
  • Prospective input: Under spatial-only backpropagation, prospective inputs remove the explicit (h/τ)^(L−ℓ) attenuation factor and change each spatial-hop prefactor from O(h/τ) to O(1).The result is a theoretical correction to the discretization-dependent gradient path, not a claim that all depth-dependent attenuation disappears.
  • Prospective input: Prospective-input coding replaces each layer’s instantaneous bottom-up drive with a parameter-free two-tap correction that leaves the recurrent transition and parallel scan unchanged.The correction applies to diagonal SSMs as well as RQFs.
  • Evidence and scope: Prospective-input coding is motivated by prospective firing and, under full BPTT, produces higher mean test accuracy than matched controls in every tested configuration.The paper treats spatial-only backpropagation as a diagnostic regime rather than a prerequisite or biologically complete training procedure.

2 Recursive Quadrature Filters (RQFs)

RQFs are biologically motivated complex-valued filters derived from an energy model and represented as constrained diagonal SSMs. Their learnable frequency and bandwidth parameters produce stable band-pass dynamics that can be discretized efficiently for deep sequence models.

  • RQF dynamics: RQFs arise from a leaky-integrator energy model that blends bottom-up input with a phase-rotation-based recurrent prediction.The parameter γ controls the balance between input tracking and recurrence, while the recurrent prior models phase evolution.
  • RQF dynamics: Each RQF is a stable scalar complex filter with learnable phase-prior frequency ω and bandwidth γ.Its pole is λ = −α + ιω0, with α = γ/τ and ω0 = (1 −γ)ω; stability holds for γ ∈(0, 1).
  • Multi-layer architecture: Deep RQF networks stack parallel scalar filters whose complex states support non-holomorphic activations such as split-ReLU.The activation need only be real-differentiable in the real and imaginary components; complex differentiability is not required.
  • Discretization: ZOH discretization integrates the linear dynamics exactly, remains unconditionally stable for left-half-plane poles, and differs from forward Euler by O((h/τ)2).The resulting coefficients depend only on filter parameters, the base time constant, and the step size, so they can be precomputed.
  • Diagonal SSM relationship: RQF layers are constrained diagonal complex SSMs that retain parallel-scan and FFT-convolutional training algorithms.The RQF constraint couples input gain and decay timescale, while general SSM output projections and residual connections remain admissible.

3 Prospective-input coding in multi-layer RQFs

Prospective-input coding advances each layer’s bottom-up signal before inter-layer mixing, while leaving the recurrent transition and scan unchanged. The correction adds a two-tap input path and is designed to reduce depth-dependent spatial-gradient attenuation.

  • Input coding: Prospective-input coding replaces ρ(y^(ℓ−1)) with its τ-step look-ahead before multiplication by W^(ℓ), unlike the instantaneous-input baseline.The operator uses χτ(z(t)) = z(t) + τ ż(t) to extrapolate a smooth signal approximately one time constant forward.
  • Discretization: The prospective discretization is a first-order IIR recurrence with a two-tap feedforward input path.Its signed previous-input tap distinguishes it from convex endpoint interpolation schemes such as Mamba-3.
  • Computational cost: Prospective input adds no trainable parameters and leaves the diagonal state transition, associative scan, and asymptotic inference and BPTT complexity unchanged.It adds only O(Tnℓ) element-wise work and a lagged read, with an O(nℓ) rolling buffer for streaming.
  • Gradient path: Under spatial-only backpropagation, prospective input changes the per-hop discretization factor from O(h/τ) to O(1), removing the explicit (h/τ)^(L−ℓ) attenuation factor.This comparison concerns the discretization prefactor; the common activation, weight, and gain products remain separate.

4 Experiments1

Experiments test prospective-input coding in RQFs, S5, and ORGaNICs under full BPTT and spatial-only backpropagation. Prospective variants improve matched full-BPTT results and show stronger gains as spatial depth increases.

  • RQF results: Under full BPTT, prospective-input coding raises mean test accuracy in every matched RQF depth, width, and feature comparison.The comparisons use matched architecture, optimizer, schedule, seeds, and parameter count.
  • Gradient-credit diagnostic: Under spatial-only backpropagation, the MFCC advantage grows from 0.69 to 3.59 to 15.17 points as depth increases.Prospective inputs also produce larger spatial per-hop gains and early-layer weight-gradient norms.
  • Speech Commands: 96.09 ± 0.23% is reached by the six-layer width-32 raw-audio RQF using 31.9k parameters, versus 95.06 ± 0.99% for αP-S5 using 40.9k.The RQF is more accurate than αP-S5 at every raw-audio depth while using fewer parameters.
  • Transfer to other recurrent substrates: The correction transfers beyond linear RQFs: prospective αP-S5 beats native S5 in every tested comparison, while prospective ORGaNICs improve at selected depths and tie at six layers.ORGaNIC improvements occur at two and four layers under full BPTT and at four and six layers under spatial-only backpropagation.
  • Path-X: 83.56 ± 2.11% versus 81.63 ± 0.80% is obtained by prospective versus non-prospective six-layer RQFs on the 16,384-step Path-X task.At four layers, the corresponding accuracies are 82.19 ± 0.88% and 81.14 ± 1.00%.

5 Related work

The paper situates prospective coding within biologically motivated recurrent learning and modern state-space modeling. It distinguishes the studied spatial-credit problem from temporal vanishing gradients and related online-learning approaches.

  • Prospective coding: Prospective coding has been used in latent-equilibrium, neuronal-least-action, and generalized-latent-equilibrium models to address timing mismatch and credit assignment.These approaches use look-ahead variables, future-discounted voltages, or prospective and retrospective operators.
  • Cortical circuit models: RQFs belong to recurrent cortical models coupling dynamics, prediction, and normalization, including quadrature-based theories and dynamically normalized ORGaNICs.The related models connect feedforward, feedback, and prior drives through recurrent dynamics.
  • Credit assignment: The studied trainability problem differs from classical RNN vanishing gradients because it concerns spatial credit across layers rather than recurrent Jacobian products through time.RTRL and equilibrium-based methods address related temporal-credit issues through different mechanisms.
  • State-space models: RQFs build on diagonal state-space models that make long-sequence recurrent computation practical through efficient scans and simpler complex diagonal implementations.The surrounding SSM family includes S4, DSS, S4D, S5, and LRU.

6 Discussion

The discussion frames prospective input as a lightweight correction for deep continuous-time recurrent networks and RQFs as an efficient diagonal-SSM substrate. It also limits the claims to supported continuous-time settings and identifies temporal-credit and selectivity extensions as open directions.

  • Machine-learning takeaways: Prospective input is a parameter-free two-tap change that preserves the parallel scan, while prospective RQFs and αP-S5 outperform matched controls under full BPTT.RQFs also outperform S5 on raw audio at every depth while using fewer parameters without normalization, gating, or residual paths.
  • Computational-neuroscience takeaways: Under spatial-only backpropagation, prospective drive preserves several times more error gain across layers and enables temporally local learning on long sequences.A six-layer stack reaches 83% on 16,000-step raw audio without differentiating recurrence through time.
  • Limitations: The correction is not directly applicable to standard LSTMs and GRUs because its continuous-time derivative and intrinsic time constant lack canonical discrete-time counterparts.The proof also isolates linear RQFs under spatial-only backpropagation, while nonlinearities, residuals, normalization, input dependence, and look-ahead choices can alter gains.
  • Future work: Extending prospective input to local learning rules that retain useful temporal credit remains open because the diagnostic deliberately discards recurrent-time credit.Selective state-space extensions with input-dependent γ and ω are also proposed as future tests.

A.1 Energy-based derivation of RQFs

The RQF is derived from an energy model balancing feedforward and prospective prior targets, yielding a stable complex-valued band-pass filter. Its parameters control resonance and bandwidth, while its real form is a damped harmonic oscillator.

  • Energy-based derivation: The normalized energy balances feedforward error and prior error, and gradient descent produces the RQF relaxation dynamics.The energy is nonincreasing and strictly decreases away from stationary points.
  • State-space form: The complex RQF is equivalent to a two-dimensional real system combining isotropic decay with planar rotation.Its eigenvalues are a stable complex-conjugate pair, and its impulse response is a damped complex exponential.
  • Frequency response: The filter has unity gain at resonance and full half-power bandwidth 2α, with ω controlling resonance and γ controlling bandwidth.Small γ yields a narrow band-pass centered near the phase-prior frequency, while larger γ broadens the response toward low-pass integration.
  • Oscillator equivalence: The RQF is always underdamped for γ ∈(0, 1) and ωτ > 0, producing a decaying sinusoidal impulse response.Its magnitude response is Lorentzian, with the damping and oscillation parameters determined by the RQF parameterization.
  • Cascade bandwidth: A within-layer cascade narrows bandwidth without adding learnable parameters, but multiplies computation on the linear path.A four-stage cascade narrows bandwidth to approximately 0.435 of a single-stage filter at fixed α; the present work uses one stage per layer.

C Spatial-only backpropagation gradient derivation: full proof

Under spatial-only backpropagation, freezing recurrent states leaves a depth-dependent discretization factor that attenuates inter-layer gradients. Prospective input removes this explicit small factor, while the remaining weight and activation products can still attenuate gradients.

  • Setup: Spatial-only backpropagation freezes prior-step recurrent states and differentiates only through the within-step spatial chain.The derivation uses real-coordinate representations to handle complex states, matrices, and potentially non-holomorphic activations.
  • Instantaneous-input backward pass: Instantaneous-input spatial gradients contain an explicit factor (h/τ)^(L−ℓ), producing discretization-induced attenuation with depth.This factor is separate from conditioning effects caused by weights or nonlinearities.
  • Prospective backward pass: For prospective input, the inter-layer coefficient changes from (h/τ)γ to (1 + h/τ)γ, eliminating the explicit (h/τ)^(L−ℓ) prefactor.The prospective path uses a current-input term while retaining a one-step input-memory contribution in the local weight gradient.
  • Scope of the bound: The analysis provides only upper bounds, so prospective coding does not guarantee complete gradients that are O(1) or depth-independent.Weight norms, activation Jacobians, and γ factors can still attenuate with depth.
  • Gradient locality: The prospective gradient remains local in time up to the two-tap input memory and local in space once the adjacent real-coordinate error is available.Exact spatial backpropagation supplies that error through the next-layer weight matrix and activation Jacobian.

D.1 Full BPTT results with 32 filters per layer

Table 5 reports full-BPTT results for non-residual 32-filter RQF stacks on Speech Commands, while Table 6 covers matched αP-S5 and native-S5 comparisons.

  • RQF results: Table 5 evaluates non-residual RQF stacks with 32 filters per layer on Speech Commands under full BPTT.The supplied caption identifies the model width, architecture, dataset, and training regime, but not individual cell values.
  • S5 comparison: Table 6 evaluates αP-S5 against native S5 using residual S5 blocks with 32 units per layer on Speech Commands under full BPTT.The αP construction changes continuous-time input gain, adds a second tap, and uses pole clipping described in the appendix.

D.2 Direct measurements of the spatial gradient path

Direct measurements show that prospective input preserves the spatial error signal substantially better than instantaneous input in deep RQFs, especially at early layers and smaller step sizes.

  • 10,290× larger prospective gradient at layer 1 than instantaneous input under spatial-only backpropagation.The gradients are similar at layer 6 but the instantaneous gradient decays much faster toward earlier layers.
  • 0.585–0.624 per-hop error gain for prospective input across the step-size sweep, while instantaneous gain falls approximately with h/τ.The measured prospective-to-instantaneous ratio follows the exact-ZOH current-tap prediction.

E.1 RQF architecture

RQF networks stack complex-valued recurrent filters without residual connections or normalization, using learnable frequency and bandwidth parameters and either instantaneous or two-tap prospective input updates.

  • E.1 RQF architecture: Six-layer RQF stacks use widths 32 or 64, split-ReLU between layers, and no normalization, gating, dropout, bidirectionality, or residual connections.A real input projection feeds complex RQF layers, and the final state is classified through a real 2n →256 →10 MLP.
  • E.1 RQF architecture: Each RQF channel has a complex state, dimensionless frequency θ_i = τω_i, and bandwidth parameter γ_i.The implementation uses ZOH recurrence; prospective input replaces the single input coefficient with two taps while retaining the state coefficient.
  • E.1 RQF architecture: Prospective input is implemented as a scalar two-tap recurrence using the current and previous inputs, with the previous-input buffer initialized to zero.This is the channel-level form of the prospective update and does not alter the recurrent transition.
  • E.1 RQF architecture: RQF pole parameters are learned through log θ_i and log γ_i leaves.The model uses bounded frequency and bandwidth parameterizations for initialization and optimization.

E.2 ORGaNICs architecture

The ORGaNICs and S5 implementations preserve their respective recurrent architectures while applying prospective input only at the input drive, with matched instantaneous and prospective variants.

  • E.2 ORGaNICs architecture: ORGaNICs stacks 2, 4, or 6 width-64 layers with learned intrinsic time constants and identity residual additions where dimensions match.The first 20 →64 MFCC layer cannot use an identity residual, and layer normalization is disabled.
  • E.2 ORGaNICs architecture: The binary input condition selects instantaneous input when δχ = 0 and prospective input χ_τ(z(t)) when δχ = 1.The prospective variant applies a look-ahead scale to the input-drive pathway.
  • E.2 ORGaNICs architecture: The prospective ORGaNICs correction enters through a two-tap backward-difference term while gain dynamics remain otherwise identical across variants.The ordered Euler implementation updates gains first, then uses the updated gains in the principal and normalization states.
  • E.2 ORGaNICs architecture: S5 converts native-clock poles to physical input rows using the pole decay rate, tying the α gain to physical pole dynamics rather than learning it separately.For mode p, λ_eff,p = (Δ_p/h)λ_p and B_eff,p is scaled accordingly.
  • E.2 ORGaNICs architecture: The αP-S5 update uses adjacent input samples with coefficients B̄+ and B̄−, adding no parameters and leaving the recurrent transition, residual path, readout, and scan unchanged.The shared prospective horizon is τ = 5h, and the delayed input starts at zero.

E.4 Optimizer and training

Training uses matched datasets, seeds, optimization schedules, and checkpoint procedures across prospective and non-prospective models, with separate full-BPTT and spatial-only regimes.

  • E.4 Optimizer and training: Prospective and non-prospective Speech Commands pairs use identical training settings and seeds, with stratified 70/15/15 splits and validation-selected checkpoints.The 10-word subset uses standardized features and fixed MFCC extraction settings.
  • E.4 Optimizer and training: RQF Speech Commands training uses AdamW, gradient clipping at 1.0, cosine annealing, and early stopping over at most 300 epochs.Complex mixing weights, input projection, and readout use 10^-3 learning rates, while RQF pole parameters use 10^-4.
  • E.4 Optimizer and training: ORGaNICs uses AdamW with label smoothing, gradient clipping, cosine annealing, and early stopping over a maximum of 300 epochs.The learning rate is 10^-3 with weight decay 10^-4.
  • E.4 Optimizer and training: Full BPTT retains complete temporal gradients, whereas spatial-only backpropagation detaches every recurrent state after each step and applies loss at each time step.The latter isolates the spatial path without differentiating through recurrence over time.
  • E.4 Optimizer and training: Path-X width-64 RQFs use Adam, batch size 64, cross-entropy at each time step, gradient clipping, and cosine annealing for 300 epochs.The task uses the pathfinder128/curv_contour_length_14 dataset with an 80/10/10 split.
Loading 2609.04134v1…