The Recurrent Neural Tangent Kernel

• The RNTK extends NTK analysis to overparametrized RNNs and provides a kernel for sequential data.Source: §1 · pp. 1–2
• The RNTK can detect long-term dependencies when hyperparameters are properly initialized, with different choices determining how dependencies are extracted through time.Source: §1 · pp. 1–2
• The paper derives RNTKs for arbitrary sequence lengths and layer counts using ReLU and erf nonlinearities, then studies hyperparameter-dependent time-step weighting.Source: §1 · p. 2
• At infinite width, the RNTK remains constant during RNN training, simplifying training dynamics to ordinary differential equations.Source: §1 · p. 2
• For equal-length sequences, weight-tied and weight-untied RNNs converge to the same RNTK, preserving the analytical form despite recurrent weight sharing.Source: §1 · p. 2
• The RNTK outperforms classical kernels, NTKs, and finite RNNs across synthetic and 56 real-world time-series classification and regression data sets.Source: §1 · p. 2

Recurrent Neural Networks (RNNs)

• A simple RNN recursively computes pre-activation and hidden-state vectors across layers and time steps from inputs, parameters, and initial hidden states.Source: §2 · p. 2
• The RNN output is produced through a linear readout layer applied to the final recurrent representations.Source: §2 · p. 2
• Figure 1 distinguishes an RNN cell, layer, and each layer’s initial hidden state within the recurrent architecture.Source: Figure 1 · p. 3

Neural Tangent Kernel (NTK)

• The NTK measures parameter-gradient interactions between network outputs on two input sequences.Source: §2 · p. 3
• Under gradient flow, parameter and output evolution is governed by the empirical NTK and the loss gradient.Source: §2 · p. 3
• For infinite-width networks, the NTK stays constant during training, reducing mean-square-error dynamics to linear ordinary differential equations equivalent to zero-ridge kernel regression.Source: §2 · p. 3

3.1 RNTK for an Infinite-Width RNN at Initialization

• The RNTK has an analytical infinite-width form for multilayer RNNs with ReLU and erf nonlinearities, including inputs of arbitrary lengths.The limiting kernel is obtained under NTK initialization and supports point-wise evaluation.Source: §3.1 · pp. 4–5
• The RNTK directly compares sequences of different lengths without zero padding, unlike common classical kernels and current NTKs.This capability follows from the theorem’s formulation for potentially different sequence lengths.Source: §3.1 · p. 4

3.2 RNTK for an Infinite-Width RNN during Training

• The RNTK remains constant during infinite-width gradient-flow training, reducing the training dynamics to a kernel-governed system under the theorem’s assumptions.The result assumes positive-definite training-kernel data and a learning rate below the stated threshold.Source: §3.2 · p. 5

3.3 RNTK for an Infinite-Width RNN Without Weight Sharing

• For equal-length inputs, infinite-width weight-tied and weight-untied RNNs converge to the same RNTK.Figure 2 also reports convergence of both finite-width variants toward the limiting kernel as width increases, with similar rates.Source: Figure 2 · p. 5, §3.3 · p. 6

3.4 Insights into the Roles of the RNTK’s Hyperparameters

• Across 56 real-world time-series classification data sets, the RNTK outperforms classical kernels, the NTK, and trained RNNs across all reported metrics.Table 1 summarizes the aggregate comparison, while detailed per-data-set results are provided elsewhere in the paper.Source: Table 1 · p. 7
• Hyperparameters and activation functions control which time steps influence the RNTK, with ReLU generally producing more stable temporal sensitivity than erf.Changes in σw can create first- or last-step oversensitivity, while nonzero σh reduces the importance of early time steps.Source: Figure 3 · p. 6, §3.4 · p. 7
• The sensitivity analysis recommends choosing hyperparameters for roughly constant time sensitivity without prior data knowledge, or targeting desired time steps when prior knowledge exists.The sensitivity s(t) measures the RNTK’s responsiveness to the input at time t.Source: §3.4 · pp. 6–7

Time Series Classification

• RNTK outperforms classical kernels, NTKs, and trained RNNs across all metrics on 56 time-series classification datasets.The study uses C-SVMs for kernels and gradient-descent training for finite RNNs.Source: §4 · p. 7
• Finite RNNs and GRUs perform poorly in the classification setting because the considered datasets are relatively small.The authors identify comparison on larger datasets as a future direction.Source: §4 · pp. 7–8

Time Series Regression

• RNTK regression gains increase with input-length variation, reflecting its ability to handle variable-length sequences naturally.The comparison uses kernel ridge regression on synthetic sinusoid and Google stock data.Source: §4 · p. 8
• RNTK maintains performance with small training sets, unlike the other compared methods in the variable-length regression experiments.Figure 4d varies training-set size at fixed Tvar = 10.Source: §4 · p. 8, Figure 4 · p. 8
• High noise can disadvantage RNTK because recurrence may propagate and amplify noise across time steps.The figure caption similarly notes better performance at low input-noise levels.Source: §4 · p. 8, Figure 4 · p. 8
• On Google stock data, the previous-time-step predictor performs best, while RNTK is the closest kernel method and improves as training samples increase.The result is attributed to the data relying heavily on recent observations.Source: §4 · p. 8

5 Conclusions

• Simple infinite-width RNNs converge to the same RNTK at initialization, after training, and with or without weight sharing.The conclusion connects this convergence to insights about variable-length inputs, training dynamics, and sensitivity to nonlinearities and initialization.Source: §5 · p. 9
• RNTK achieves superior time-series classification and regression performance compared with classical kernels, the NTK, and trained RNNs.The authors identify gated-RNN RNTKs and extensions to finite RNNs as future research directions.Source: §5 · p. 9

A.1 Time series classification

• Table 2 reports test accuracy for each model across 56 UCR time-series classification datasets.The table caption identifies the dataset repository and evaluation target.Source: Table 2 · p. 12
• Classification compares RNTK, RBF, polynomial, NTK, and finite RNN variants using C-SVMs, cross-validation, and reported aggregate metrics.The metrics include average accuracy, P90, P95, PMA, and Friedman Rank.Source: §A.1 · pp. 13–14
• The classification setup includes sequence lengths of 50, 100, 200, and 500, with selected layer and variance hyperparameter ranges.The RNTK uses a single ReLU layer in the described settings.Source: §A.1 · pp. 13–14
• The appendix also defines P90, P95, PMA, and Friedman Rank as dataset-level performance summaries relative to maximum accuracy or classifier ranking.Higher P90 and PMA and lower Friedman Rank indicate better performance.Source: §A.1 · p. 14

A.2 Time Series Regression

• Regression uses five-fold validation and kernel ridge regression with a selected ridge term for the time-series regression experiments.The supplied passage gives the validation and estimator choice but not the resulting scores.Source: §A.2 · p. 14

B.1 Preliminary: Netsor programs

• Weight sharing breaks the usual sequential-width Gaussian-process proof because conditioning on prior layers does not leave independent weights.The framework uses a Gaussian conditioning trick to handle recurrent weights.Source: §B.1 · pp. 14–15
• Netsor programs encode neural computations through Gaussian weight variables, linear combinations, matrix multiplication, and coordinate-wise nonlinearities.A-vars represent weights, G-vars pre-activations, and H-vars post-activation vectors.Source: §B.1 · p. 15
• The GP covariance is computed recursively from input-dependent quantities and previously introduced variables.The recursion uses Gaussian expectations over functions of the G-vars.Source: §B.1 · pp. 15–16
• The stated GP-limit results assume nonlinearities bounded by e^(cx^2−α) for some α > 0 and differentiable.This is an explicit regularity condition for the preliminary results.Source: §B.1 · p. 16
• NTK initialization factors the weight standard deviation so forward computation is unchanged while backward computation is normalized.Without the corresponding scaling, the RNTK can acquire an n-dependent factor.Source: §B.1 · p. 16

B.2 Proof for Theorem 1: Single layer Case

• The single-layer derivation computes forward and backward pass kernels recursively, using Gaussian independence and zero cross-terms under the stated rules.The backward-pass calculation relies on Corollary 1, which treats forward and backward weights as identically distributed independent copies under its assumptions.Source: §B.2 · pp. 16–18
• The RNTK is obtained by summing gradient contributions from recurrent weights, input weights, biases, and output weights in the infinite-width limit.The derivation uses the vectorized parameter set and replaces inner products with expectations before simplifying the resulting kernel.Source: §B.2 · pp. 18–19
• For multidimensional outputs, independent output weights make off-diagonal kernel terms zero, while diagonal terms equal the single-output kernel.Thus each output coordinate uses the single-layer result independently.Source: §B.2 · pp. 19–20
• The proof concludes by establishing the single-layer formula for the RNTK.The stated theorem follows after the forward, backward, and parameter-gradient terms are combined.Source: §B.2 · p. 20

B.3 Proof for Theorem 1: Multi-Layer Case

• The multi-layer derivation recursively extends the forward and backward kernels across layers, with nonzero backward terms aligned by the sequence-length offset τ.The recursion continues through hidden layers until it reaches a terminal backward-pass term.Source: §B.3 · pp. 20–21
• Forward kernels for intermediate layers use newly sampled weights at each layer, while the backward kernels are computed from the final layer through the remaining layers.The proof separately handles last-layer and other-layer terms before assembling the full kernel.Source: §B.3 · pp. 20–21
• The multi-layer RNTK combines recurrent-weight, input-weight, bias, and output-weight gradient contributions across all layers and time steps.The resulting expression is simplified by summing the terms and replacing vector inner products with expectations.Source: §B.3 · pp. 21–22
• The proof concludes the multi-layer theorem and extends the construction to multidimensional outputs.The multidimensional expression is obtained after the single-output derivation.Source: §B.3 · p. 22

B.4 Proof for Theorem 3: Weight-Untied RNTK

• For equal-length sequences, the weight-untied forward kernel follows the same recursion as the weight-tied RNN.For different-length sequences, the forward kernel becomes zero because the corresponding time steps use different weights.Source: §B.4 · p. 22
• The untied RNTK is assembled from recurrent-weight, input-weight, bias, and output-weight gradient terms for aligned time steps.These terms are written using expectations of hidden-state and backward-gradient products.Source: §B.4 · pp. 22–23
• Different weights at unequal time steps force the corresponding backward-pass kernel Π(t,t′)(x, x′) to zero.The remaining calculations therefore match the weight-tied case when τ = 0.Source: §B.4 · pp. 22–23
• The proof concludes that the weight-untied construction agrees with the weight-tied RNN when τ = 0.This establishes the stated weight-untied result.Source: §B.4 · p. 23

C Proof for Theorem 2: RNTK Convergence after Training

• The convergence theorem assumes local Jacobian Lipschitzness, probabilistic convergence of the empirical NTK at initialization, and positive definiteness on the input set.Under these assumptions and a stated learning-rate condition, gradient flow satisfies the theorem’s high-probability conclusion.Source: C Proof for Theorem 2: RNTK… · p. 23
• The proof extends RNTK convergence analysis by showing that a multi-layer RNN Jacobian is locally Lipschitz with high probability.This is the central additional property used in the convergence argument.Source: Proof: See [29] · p. 24

Proof: See [29]

• The proof uses induction over layers together with Gaussian random-matrix and random-vector concentration lemmas.The induction bounds forward quantities and first-layer gradients, while boundedness and Lipschitz assumptions on φ and φ′ control the nonlinearities.Source: Proof: See [29] · p. 24
• The resulting bound depends on σw, σu, σb, the number of layers L, the sequence length T, and the input norms.A union-bound argument extends the estimates from individual points to the entire dataset.Source: Proof: See [29] · pp. 25–26