On Explaining the Surprising Success of Reservoir Computing Forecaster of Chaos? The Universal Machine Learning Dynamical System with Contrasts to VAR and DMD

Erik Bollt

Source-linked AI summary

On Explaining the Surprising Success of Reservoir Computing Forecaster of Chaos? The Universal Machine Learning Dynamical System with Contrasts to VAR and DMD

Erik Bollt

arXiv:2008.06530v6Aug 14, 2020physics.data-anmath.DS

TL;DR

The paper addresses why reservoir computing can forecast complex dynamical systems despite random internal weights. It analyzes a linearized RC, connects it to VAR, NVAR, and DMD, and reports reasonable short-term forecasting and strong performance for quadratic read-out. These conclusions are bounded by assumptions about data representativeness and the applicability of Wold-based representations.

Problem
The paper asks how RC can forecast broadly despite randomly selected input and internal weights, given the limited theoretical explanations of this success.
Method
The paper simplifies RC with linear activation, analyzes its read-out fitting, and connects the resulting formulation to VAR, Wold theory, NVAR, and DMD.
Results
The linear RC is connected to VAR with reasonable short-term forecasts, while a linear reservoir with quadratic read-out is equivalent to a quadratic NVAR that performs quite well.
Takeaways & Limitations
RC’s forecasting behavior can be interpreted through established time-series and dynamical-mode-decomposition formulations rather than as unexplained performance from random weights.
Takeaways & Limitations
Wold-based representation claims depend on assumptions that may not fit the colored or otherwise non-white noise in chaotic dynamical-system data.

Abstract

from arXiv · show

Machine learning has become a widely popular and successful paradigm, including in data-driven science and engineering. A major application problem is data-driven forecasting of future states from a complex dynamical. Artificial neural networks (ANN) have evolved as a clear leader amongst many machine learning approaches, and recurrent neural networks (RNN) are considered to be especially well suited for forecasting dynamical systems. In this setting, the echo state networks (ESN) or reservoir computer (RC) have emerged for their simplicity and computational complexity advantages. Instead of a fully trained network, an RC trains only read-out weights by a simple, efficient least squares method. What is perhaps quite surprising is that nonetheless an RC succeeds to make high quality forecasts, competitively with more intensively trained methods, even if not the leader. There remains an unanswered question as to why and how an RC works at all, despite randomly selected weights. We explicitly connect the RC with linear activation and linear read-out to well developed time-series literature on vector autoregressive averages (VAR) that includes theorems on representability through the WOLD theorem, which already perform reasonably for short term forecasts. In the case of a linear activation and now popular quadratic read-out RC, we explicitly connect to a nonlinear VAR (NVAR), which performs quite well. Further, we associate this paradigm to the now widely popular dynamic mode decomposition (DMD), and thus these three are in a sense different faces of the same thing. We illustrate our observations in terms of popular benchmark examples including Mackey-Glass differential delay equations and the Lorenz63 system.

1 Introduction

The paper asks why reservoir computing can forecast complex dynamical systems despite randomly selected input and internal weights. It simplifies RC with linear activation to connect it with VAR, Wold theory, NVAR, and DMD.

1 Introduction: RNNs suit temporal forecasting but can be expensive to optimize because they contain many trainable parameters and nonlinear optimization.RC addresses this training burden by training only its output layer with least squares.
1 Introduction: RC trains only output weights using efficient least squares, while input and internal weights are selected randomly.This simplification makes RC cheaper than fully trained recurrent networks.
1 Introduction: The paper investigates how RC remains successful despite randomly selected read-in and inner weights, rather than proposing a new forecasting method.The linear-activation variant sacrifices some fidelity to enable a detailed theoretical analysis.
1 Introduction: Linear activation makes RC a linear recursion whose read-out weights can be fitted as a well-posed problem despite random read-in and inner weights.The paper connects this formulation to VAR and the representation theory associated with the Wold theorem.
1 Introduction: A linear reservoir with quadratic read-out is equivalent to a quadratic NVAR containing all monomial quadratic terms.The paper reports that this formulation performs quite well.
1 Introduction: The paper relates RC, VAR, NVAR, and DMD as connected formulations for understanding dynamical-system forecasting.The examples compare nonlinear and linear RC variants on Mackey-Glass and Lorenz63 systems.

2 The Data as Sampled From a Stochastic Process

The paper frames forecasting data as ordered samples from deterministic or stochastic processes and uses Mackey-Glass and Lorenz63 as benchmark systems. It emphasizes that forecast quality depends strongly on representative, sufficiently long data.

2 The Data as Sampled From a Stochastic Process: Figure 1 uses N = 10, 000 Mackey-Glass data points as the training set and displays delay coordinates with τ = 20.The figure’s top panel shows the time series; the bottom panel shows the three-dimensional projection.
2 The Data as Sampled From a Stochastic Process: Forecasting requires sampled data from a deterministic or stochastic dynamical process.The data are represented as time-indexed random variables and observations.
2 The Data as Sampled From a Stochastic Process: The framework assumes an ordered sample of vector-valued observations, with uniform timing as a simplifying assumption.Uniform timing is expressed as a constant interval h between successive observations.
2 The Data as Sampled From a Stochastic Process: Forecast quality can be brittle because data-driven methods generally need sufficiently long, representative data and extrapolation is harder than interpolation.The paper notes that performance depends on how close forecast targets are to observed inputs.
2 The Data as Sampled From a Stochastic Process: The benchmark examples include both the Mackey-Glass differential delay equation and the Lorenz63 system.These systems are presented as classic machine-learning benchmarks.
2 The Data as Sampled From a Stochastic Process: The Mackey-Glass differential delay equation is used as a popular benchmark because it is chaotic and dimensionally complex.Figure 1 shows its time series and a three-dimensional delay-coordinate projection.

3 Review of The Traditional RC With Nonlinear Sigmoidal Activation Function

Traditional reservoir computing trains only the read-out weights while randomly selecting the input and recurrent weights, making training computationally cheap. Despite this simplification, nonlinear RC produces accurate one-step forecasts and plausible future forecasts, though performance depends on design choices.

Architecture and training: The reservoir dimension is chosen larger than the data dimension, with dr > dx, and A may be designed using spectral radius, sparsity, or echo-state considerations.The scale of randomly selected Win is controlled by γ, while regularization mitigates overfitting.
Architecture and training: RC trains only Wout using a simple least-squares computation, while Win and A are randomly selected rather than optimized.This avoids the expensive nonlinear optimization required to train a full recurrent neural network.
Forecasting behavior: For Mackey-Glass data, nonlinear RC one-time-step forecasts nearly overlay the true trajectory after training on N = 10,000 samples.The example uses dr = 500 and regularization λ = 1.e−6 in the figure caption.
Forecasting behavior: Closed-loop forecasts into the future diverge from the true Mackey-Glass trajectory, unlike one-time-step forecasts evaluated across the training data.The future-forecast setup feeds the RC output back into its input after training.
Forecasting behavior: Adding quadratic read-out terms yields a quadratic NVAR that the authors observe performs quite well, whereas linear RC performance is reduced but remains useful.The paper emphasizes explaining why read-out-only fitting works rather than proposing the linear variant as a superior method.

4 RC With A Fully Linear Activation, q(s) = s, Yields a VAR(k)

Replacing the nonlinear activation with q(s) = s makes the reservoir state an explicit linear combination of delayed inputs. The resulting linear RC is equivalent to a VAR(k), connecting random reservoir parameters to fitted autoregressive coefficients and established time-series theory.

Linear RC and VAR connection: With identity activation q(s) = s, the RC becomes equivalent to a vector autoregressive process VAR(k).This simplification preserves useful forecasting while enabling analysis through VAR representation theory and the Wold theorem.
Coefficient decomposition: The VAR coefficient matrices are a_j = Wout A^{j−1} Win, so random reservoir matrices determine the autoregressive coefficients through their iterated projections.Each coefficient is a dx × dx matrix formed by multiplying the read-out, recurrent, and input transformations.
Coefficient decomposition: For exactly k VAR matrices, the random reservoir degrees of freedom collapse onto the finitely many fitted parameters of a_1, …, a_k.For longer observations, the randomness is represented through matrices A^{j−1}Win across multiple delays.
Example: The fully linear RC forecasts Mackey-Glass data into the future less accurately than the nonlinear RC, but still produces forecasts.The comparison is presented as evidence for the theoretical relationship and read-out fitting, not as a performance claim for the linear method.
Linear RC and VAR connection: The reservoir state rℓ+1 is a finite sum of delayed inputs transformed by powers of A and the random input matrix Win.The recursion exposes how memory enters through Aℓ−1Win, Aℓ−2Win, and subsequent terms.
Coefficient decomposition: Wout is sufficient to fit the model once A and Win are fixed, because the read-out estimates projections of VAR coefficients into the reservoir-induced row space.The paper derives this relationship using regularized least squares and pseudo-inverse identities.

5 VAR(k) Theory Suggests Convergence with k

The linear RC connects naturally to VAR representations through Wold-theorem conditions and invertibility. Finite-order VAR fitting provides a practical least-squares interpretation, while stability and fading-memory conditions govern convergence.

A covariance-stationary process admits a Wold VMA representation with a deterministic component and square-summable coefficient matrices.
When the delay polynomial is invertible, the WMA representation yields a VAR representation expressing current values as linear combinations of prior values.
Finite-order VAR(k) models approximate VAR(∞) representations as k increases because the coefficient matrices are square summable.
Least-squares estimation of VAR(k) directly relates to the read-out fitting procedure in the linear reservoir computer.
The Wold-based explanation relies on stationarity and white-noise assumptions that may not hold for chaotic dynamical-system data, although regression can still work empirically.
VAR(k) stability can be analyzed through a companion VAR(1), requiring spectral radius inside the unit disc or characteristic roots outside it.

6 Quadratic Nonlinear VAR

A linear reservoir with Hadamard-quadratic read-out is equivalent to a quadratic nonlinear VAR. This extension retains least-squares read-out fitting while improving longer-range chaotic forecasting.

The quadratic read-out augments reservoir variables with componentwise products r ◦ r.
A linear VAR with Hadamard-quadratic read-out is equivalent to a quadratic nonlinear VAR containing all quadratic terms in delayed variables.
The generalized read-out remains obtainable through a least-squares pseudoinverse fit.
Linear-reservoir quadratic-read-out forecasting is better and longer range than linear-reservoir linear-read-out forecasting, while preserving chaotic-attractor statistics after pointwise errors grow.

7 Is There a Connection to DMD-Koopman?

The paper identifies a connection among linear RC, VAR, and DMD because each represents linear evolution from time-delay data. The connection is structural rather than complete, since their fitted matrices impose different freedoms.

Identity-activation RC produces a linear process related to VAR, while DMD models finite approximations of Koopman-operator action on observables.
Time-delay VAR evolution can be written in an exact-DMD form using data matrices X and X′ and a DMD matrix K.
Exact DMD is popular for its simple implementation and interpretation through modal behaviors.
The VAR companion matrix A and exact-DMD matrix K both encode X′ as a linear function of X, but their least-squares parameter freedoms differ.
Arnoldi-style comparisons relate eigenvalues and eigenvectors of the DMD companion matrix and reservoir/VAR companion matrix, while VAR uses matrix-valued coefficients rather than scalar coefficients.

8 Examples

The paper illustrates its RC, VAR, and NVAR observations on Mackey–Glass and Lorenz63 benchmarks. On Lorenz63, nonlinear and quadratic-readout variants preserve attractor-like behavior better than the fully linear model.

Mackey–Glass: Mackey–Glass is used as a benchmark because its attractor complexity varies with parameters despite arising from an infinite-dimensional delay system.
Lorenz63: Lorenz63 provides a familiar chaotic-attractor benchmark for illustrating the paper’s forecasting concepts.
Lorenz63: The fully nonlinear RC forecasts Lorenz63 excellently and continues producing a Lorenz-like attractor after errors accumulate.
Lorenz63: The fully linear RC is equivalent to VAR and gives good short-term forecasts, but later forecasts converge toward zero instead of a Lorenz-like attractor.
Lorenz63: The linear RC with quadratic read-out forecasts well and reproduces the attractor comparably to the fully nonlinear RC after errors grow.

9 On Fading Memory

The paper relates reservoir memory to decaying VAR coefficient matrices and examines this behavior in Mackey–Glass and Lorenz systems. Theoretical decay is governed by the reservoir matrix spectrum, while numerical experiments show system-dependent fading memory and agreement between RC-derived and directly fitted VAR coefficients.

9.1 Fading Memory In Terms of the Role of the Internal Layers: Lorenz forecasts distinguish linear and nonlinear read-outs: fully linear RC forecasts lose the attractor, whereas quadratic read-outs preserve a Lorenz-like attractor after errors grow.The fully linear RC is equivalent to a VAR and provides good forecasts only for a finite, shorter interval.
9.1 Fading Memory In Terms of the Role of the Internal Layers: Reservoir coefficient matrices a_j are bounded by products involving Wout, A^(j−1), and Win, linking memory decay to powers of the internal matrix A.The induced-norm bound is ∥a_j∥⋆ ≤ ∥Wout∥⋆∥A∥^(j−1)∥Win∥⋆.
9.1 Fading Memory In Terms of the Role of the Internal Layers: When ρ(A) < 1 and Wout is bounded, the coefficient matrices converge componentwise to zero as the lag increases.The convergence need not be monotone, and the bound need not be sharp.
9.2 Experiments on Fading Memory of the RC: Mackey–Glass coefficients become negligible by k ≥ 6, indicating a descriptive VAR(6) and pronounced fading memory.The system has a relatively small positive maximum Lyapunov exponent, λmax = 0.0058.
9.2 Experiments on Fading Memory of the RC: Lorenz coefficients decrease to small values by k = 9, but larger-lag estimates fail to continue decreasing because long-range chaotic forecasts are difficult.The early coefficients are stable across samples, whereas tail values vary; the RC-derived and directly fitted VAR curves almost coincide.

10 Conclusion

The conclusion emphasizes reservoir computing as a computational shortcut for dynamical forecasting: only the output layer is trained, while input and internal weights are random. The paper interprets this success through connections to VAR and NVAR representations, with nonlinear read-outs supporting longer and more attractor-faithful forecasts.

10 Conclusion: Reservoir computing avoids full recurrent-network optimization by randomly selecting input and internal weights and fitting only the output layer by least squares.This reduces the training problem from nonlinear high-dimensional optimization to linear-algebraic fitting.

13 Appendix: Review of Regularized Pseudo-Inverse

The appendix reviews SVD-based Tikhonov regularization for computing a stable pseudo-inverse. A small positive regularization parameter prevents unstable division by near-zero singular values and mitigates overfitting.

13 Appendix: Review of Regularized Pseudo-Inverse: The SVD X = UΣVᵀ yields the ridge-regression pseudo-inverse X†_λ = (XᵀX + λI)^−1Xᵀ, equivalently expressed through singular values.The construction applies to rectangular matrices and is reviewed for stable least-squares fitting.
13 Appendix: Review of Regularized Pseudo-Inverse: A small positive λ regularizes near-zero singular values, reducing overfitting and avoiding unstable denominators in the pseudo-inverse.Unless otherwise stated, the paper uses λ = 1.0 × 10^-8.

14 Appendix: On Quadratic NVAR Connection to RC

The appendix derives the connection between a linear reservoir with Hadamard quadratic read-out and a quadratic nonlinear VAR. It expands quadratic reservoir terms into linear and pairwise lagged input features, showing that the trained read-out supplies the corresponding NVAR coefficients.

14 Appendix: On Quadratic NVAR Connection to RC: A linear reservoir with Hadamard quadratic read-out corresponds to a quadratic NVAR, extending the paper’s linear VAR connection.The quadratic read-out uses both reservoir variables r and componentwise products r ◦ r.
14 Appendix: On Quadratic NVAR Connection to RC: The augmented read-out matrix R has 2dr rows, combining linear reservoir features with their componentwise squares.Wout is partitioned into top and bottom blocks that act on the linear and quadratic NVAR terms.
14 Appendix: On Quadratic NVAR Connection to RC: The matrix P2(B,B) converts Hadamard products of Bw into ordinary matrix multiplication using a matrix of pairwise column products.This identity allows the quadratic read-out to be written in explicit matrix form.
14 Appendix: On Quadratic NVAR Connection to RC: The quadratic data matrix X2 contains pairwise quadratic combinations of lagged observations, while the resulting coefficient matrices generalize linear VAR coefficients.The construction uses p2(x_i,x_j) features across lagged data columns.
14 Appendix: On Quadratic NVAR Connection to RC: Quadratic reservoir expansions generate cross terms involving delayed inputs, represented through operators such as P2(Win, AWin) and P2(Win, Win).These terms collect pairwise products across lagged linear-reservoir contributions.