PaperlayerView exact revision on arXiv

Source-linked AI summary

Next Generation Reservoir Computing

Daniel J. Gauthier, Erik Bollt, Aaron Griffith, Wendson A. S. Barbosa

arXiv:2106.07688v2cs.LGnlin.AO

TL;DR

Reservoir computing relies on random matrices and many metaparameters, while nonlinear vector autoregression offers an equivalent formulation without a reservoir. This paper demonstrates that this next-generation approach excels across reservoir-computing benchmarks using shorter training data sets and training time.

  • Problem

    Traditional reservoir computing depends on randomly sampled matrices and multiple metaparameters whose effective selection lacks known design rules.

  • Method

    The paper replaces the reservoir with nonlinear vector autoregression using time-delay observations and nonlinear features trained through regularized linear regression.

  • Results

    NG-RC performs at a state-of-the-art level across forecasting, long-term climate reproduction, and unseen-data inference benchmarks while using exceedingly small data sets.

  • Takeaways & Limitations

    Nonlinear vector autoregression provides a next-generation reservoir computer that avoids direct random-matrix and metaparameter difficulties while retaining strong benchmark performance.

  • Takeaways & Limitations

    The study considers only data generated by noise-free numerical simulations, with noise-tolerance evidence reported for Lorenz63 in supplementary materials.

Abstract

from arXiv · show

Reservoir computing is a best-in-class machine learning algorithm for processing information generated by dynamical systems using observed time-series data. Importantly, it requires very small training data sets, uses linear optimization, and thus requires minimal computing resources. However, the algorithm uses randomly sampled matrices to define the underlying recurrent neural network and has a multitude of metaparameters that must be optimized. Recent results demonstrate the equivalence of reservoir computing to nonlinear vector autoregression, which requires no random matrices, fewer metaparameters, and provides interpretable results. Here, we demonstrate that nonlinear vector autoregression excels at reservoir computing benchmark tasks and requires even shorter training data sets and training time, heralding the next generation of reservoir computing.

I. Introduction · II. Background: Reservoir Computing

Reservoir computing efficiently learns complex dynamical systems but relies on random matrices and metaparameter optimization. The paper presents NG-RC, an equivalent nonlinear vector autoregression that avoids a reservoir while achieving state-of-the-art benchmark performance with very small data sets.

  • I. Introduction: Data-hungry machine-learning methods require long observations and substantial computational resources for modeling dynamical systems.
  • I. Introduction: Reservoir computing is well-suited to learning dynamical systems, including chaotic and complex spatio-temporal behaviors.
  • I. Introduction: Traditional RCs use randomly assigned input and recurrent weights, training only output weights through regularized linear optimization.The reservoir contains interconnected neurons that process observed data into weighted output states.
  • I. Introduction: Random matrices can perform inconsistently, while several performance-sensitive RC metaparameters require optimization and lack clear design rules.
  • I. Introduction: An RC with linear reservoir nodes and nonlinear functions in its feature vector is an equally powerful universal approximator and is mathematically identical to an NVAR.The NVAR feature vector uses time-delay observations and nonlinear functions, eliminating the need for a reservoir.
  • I. Introduction: NG-RC demonstrates easy design and state-of-the-art performance on short-term forecasting, chaotic-system climate reproduction, and inference from unseen data.It uses exceedingly small data sets and avoids the random and parametric difficulties of directly implementing traditional RC.
  • II. Background: Reservoir Computing: An RC broadcasts input data into an N-node recurrent network, whose state is transformed into an output through a feature vector and output-weight matrix.The connectivity matrix A and input coefficients W are randomly chosen and kept fixed; node dynamics use decay, activation, and bias terms.
  • II. Background: Reservoir Computing: Standard RCs commonly use tanh activation, a linear feature vector equal to the reservoir state, and supervised regularized least-square regression.The ridge parameter α prevents overfitting to the training data.

Linear reservoir + Nonlinear output

Linear reservoirs can shift nonlinearity from the recurrent reservoir to the output feature vector while retaining universal approximation power. A squared-node feature expansion provides a simple example, with performance comparable to standard reservoir computing.

  • Linear reservoir + Nonlinear output: Moving nonlinearity from the reservoir to the output layer yields an equivalently powerful universal approximator.The reservoir nodes use the linear activation f(r)=r, while the total feature vector becomes nonlinear.
  • Linear reservoir + Nonlinear output: Including squared values of all reservoir nodes creates a nonlinear feature vector through the Hadamard product r⊙r.The nonlinear feature vector concatenates the original reservoir state with its elementwise square.
  • Linear reservoir + Nonlinear output: A linear reservoir with nonlinear output shows comparable performance to standard reservoir computing.The output construction uses vector concatenation, denoted by ⊕.

III. The NG-RC · Linear features · Nonlinear features

The NG-RC constructs features directly from sampled input data without a neural network, combining constant, linear, and nonlinear components trained with Tikhonov regularization. Its linear features use delayed observations, while low-order polynomial nonlinearities provide high prediction ability and establish equivalence with polynomial-readout linear reservoir computers.

  • III. The NG-RC: The NG-RC creates a feature vector directly from discretely sampled input data without requiring a neural network.Its total feature vector combines a constant, linear component, and nonlinear component, with outputs trained using Tikhonov regularization.
  • Linear features: Linear features concatenate the d-dimensional input at the current time and k-1 previous times, spaced by s, producing d k components.For driven systems or systems with adjustable accessible parameters, the drive signal or parameters are also included.
  • Linear features: Small k values are sufficient in practice because the Volterra series converges rapidly and truncation therefore causes little error.This choice is also motivated by the accuracy of multistep numerical integration using only a few sub-intervals.
  • Linear features: The NG-RC requires only s k warm-up time steps to form the first feature vector, substantially less than traditional reservoir computers typically require.With s=1 and k=2, only 2 warm-up data points are needed.
  • Nonlinear features: Nonlinear features are nonlinear functionals of the linear features, with polynomials serving as a natural starting point because they underpin Volterra representations.The paper finds that low-order polynomials are enough to achieve high performance.
  • Nonlinear features: Quadratic nonlinear features capture all monomials through the outer product of the linear feature vector with itself, yielding (d k)^2 elements before retaining unique terms.Unique monomials are collected using the ⌈⊗⌉ operator; a p-order feature repeats the linear feature vector p times.
  • Nonlinear features: NVAR is mathematically equivalent to a linear reservoir computer with polynomial nonlinear readout, while the NG-RC is more efficient and requires fewer optimized metaparameters.Each NVAR implicitly defines the traditional reservoir’s connectivity matrix and other parameters, and every linear polynomial-readout reservoir computer can be expressed as an NVAR.

IV. Model Systems and Tasks

The study generates forecasting and inference data by numerically integrating the Lorenz 1963 weather-system model, a three-variable nonlinear dynamical system. This system exhibits deterministic chaos and has a Lyapunov time of 1.1 time units.

  • Lorenz63: Training and testing data for one forecasting task and the inference task are generated by numerically integrating Lorenz’s 1963 weather-system model.The model consists of three coupled nonlinear differential equations.
  • Lorenz63: The state X(t) comprises three Rayleigh-Bénard convection observables and evolves chaotically with sensitive dependence on initial conditions.Its phase-space trajectory forms a strange attractor.
  • Lorenz63: 1.1 time units is the Lyapunov time of the system, characterizing the divergence timescale for its chaotic dynamics.The paper refers to this system as Lorenz63.

Double-scroll electronic circuit · Forecasting task

The NG-RC forecasts double-scroll dynamics through one-step-ahead prediction, learning an autonomous flow during testing. This nonpolynomial, odd-symmetric system tests whether polynomial feature vectors can reproduce dynamics without relying on truncated Taylor expansions.

  • Double-scroll electronic circuit: The double-scroll circuit is governed by a three-variable nonlinear system with ΔV = V₁ − V₂.Its governing equations are given explicitly in equation (8).
  • Double-scroll electronic circuit: The selected parameters produce a Lyapunov time of 7.81 time units.The parameters are R₁ = 1.2, R₂ = 3.44, R₄ = 0.193, α = 11.6, and Iᵣ = 2.25 × 10^-5.
  • Double-scroll electronic circuit: The circuit was chosen because its vector field is nonpolynomial and truncated sinh expansions can substantially alter the predicted attractor.Large ΔV values make the Taylor-series approximation particularly problematic.
  • Double-scroll electronic circuit: The task shows that polynomial feature vectors can work for nonpolynomial vector fields, consistent with Volterra representations of dynamical systems.This conclusion follows from the double-scroll forecasting experiment.
  • Forecasting task: The NG-RC forecasts Lorenz63 and double-scroll dynamics one step ahead, using Tikhonov-regularized output weights during listening.During testing, predicted outputs feed back into the input, making the NG-RC autonomous.
  • Forecasting task: For double-scroll forecasting, the feature design respects the attractor’s odd symmetry and zero means, yielding [d k + (d k)(d k + 1)(d k + 2)/6] components.The construction is specialized to the stated symmetry and mean properties.
  • Forecasting task: The NG-RC simultaneously learns the vector field and an efficient one-step-ahead integrator, thereby learning the dynamical system’s flow.This avoids learning the vector field and integrator separately.

Inference task … Task 2: Forecasting the double-scroll system

The NG-RC forecasts Lorenz63 and the double-scroll system from short training sets, achieving accurate predictions and attractor reconstruction with minimal computation. It also supports inference of an unobserved dynamical variable from observed variables in field-deployment settings.

  • Inference task: The inference task trains on all three Lorenz63 variables but tests whether z can be inferred using only x and y.This setup represents applications where a dynamical variable is measured in the laboratory but unavailable during field deployment.
  • V. Results: The Results section demonstrates NG-RC performance on forecasting and inference tasks using short time-series training data.The supplied results cover Lorenz63 forecasting, double-scroll forecasting, and missing-variable inference.
  • Task 1: Forecasting Lorenz63: 400 data points per variable and <10 ms of single-core training compute yielded Lorenz63 training with NRMSE=1.06±0.01×10^-4.The Lorenz63 setup used dt=0.025, k=2, and s=1.
  • Task 1: Forecasting Lorenz63: Lorenz63 forecasting achieved NRMSE=2.40±0.53×10^-3 over one Lyapunov time during the forecasting phase.The predicted and true datasets remained similar, and forecasts extended well out to ~5 Lyapunov times.
  • Task 1: Forecasting Lorenz63: The NG-RC reproduced Lorenz63’s long-term climate and matched the predictive ability of optimized traditional RCs using 100’s to 1,000’s of reservoir nodes.The attractor reconstruction was qualitatively similar between predicted and true strange attractors.
  • Task 2: Forecasting the double-scroll system: For the double-scroll system, training was extended from 10 to 100 units, with dt = 0.25 and 62 total features from cubic monomials.The configuration used d=3, k=2, and s=1, while retaining 400 data points for each variable.
  • Task 2: Forecasting the double-scroll system: Double-scroll forecasting achieved NRMSE=4.5±1.0×10^-3, with predictive ability similar to the Lorenz63 task.The figure compares true and predicted datasets during the forecasting phase and shows true and predicted strange attractors.

Task 3: Inferring unseen Lorenz63 dynamics · VI. Discussion

Task 3 shows that NG-RC can infer an unseen Lorenz63 component from only 400 training points. The discussion attributes its computational advantages to smaller feature vectors, shorter training and warmup, and fewer metaparameters, while noting limitations concerning noise, dimensionality, and feature selection.

  • Task 3: Inferring unseen Lorenz63 dynamics: 400 training points are sufficient for good performance when NG-RC infers an unseen Lorenz63 component using k=4, s=5, and dt=0.05.The setup embeds the full attractor to infer the other component, as informed by Takens’ embedding theorem.
  • Task 3: Inferring unseen Lorenz63 dynamics: NRMSE=9.5±0.1×10-3 during training and NRMSE = 1.75±0.3×10-2 during testing demonstrate close agreement between NG-RC predictions and Lorenz63 data.The figure reports α=0.05 for the training result.
  • VI. Discussion: NG-RC is computationally faster than traditional RC because its smaller feature vector yields fewer adjustable parameters.The discussion attributes reduced training-data requirements to having fewer fit parameters.
  • VI. Discussion: NG-RC combines features of NARX methods with modern regression methods, while differing from NARX through Tikhonov regularization and a theoretical link to an implicit RC.The passage identifies multi-input, multiple-output NARX as the closest related approach.
  • VI. Discussion: NG-RC forecasts over finite time steps, unlike data-driven linearization methods that model vector fields through finite linear subspaces of simple functions.The related methods use least squares, LASSO, or information-theoretic procedures to simplify models.
  • VI. Discussion: Small output-weight components suggest that features may be removable without increasing testing error, but sequential NARX term selection is tedious and ignores feature correlations.The NARX procedure starts with the fewest feature terms and adds terms that substantially reduce observed testing error.
  • VI. Discussion: The study uses noise-free numerical simulations and low-dimensional systems, while regularized regression is described as enabling noise tolerance and prior work suggests NG-RC may extend to complex systems.The paper reports supplementary Lorenz63 results under strong noise and highlights implications for short-data learning and digital twins.

Methods … Lorenz63 Unstable Steady States

The paper verifies Lorenz63 forecasts by comparing unstable steady states and return maps, while determining true unstable steady states from the system’s zero-derivative solutions. Accuracy is quantified using uniformly scaled L2 distances between predicted and true steady states.

  • Methods: Results were computed with Python 3.7.9, NumPy 1.20.2, and SciPy 1.6.2 on an x86-64 CPU running Windows 10.Exact numerical results, including unstable steady states and NRMSE, may vary slightly with the software used.
  • Supplementary Information for: The supplementary information accompanies the paper Next Generation Reservoir Computing.
  • Supplementary Information for: The supplementary information lists Daniel J. Gauthier, Erik Bollt, Aaron Griffith, and Wendson A.S. Barbosa as authors.
  • Forecasting Verification: Lorenz63 forecast verification compares predicted and true unstable steady states and qualitatively compares their return maps.For the double-scroll system, verification compares only predicted and true unstable steady states.
  • Lorenz63 Unstable Steady States: The true Lorenz63 unstable steady states are obtained by setting the derivatives in Eq. (7) to zero and solving for the three variables.This procedure yields three solutions.
  • Lorenz63 Unstable Steady States: The three true Lorenz63 unstable steady-state solutions include Xuss = [0,0,0]T and two nonzero solutions parameterized by β and ρ.The nonzero solutions are presented in Eq. (12).
  • Lorenz63 Unstable Steady States: 1.2±1.4×10^-3 is the L2 distance from the predicted zero unstable steady state to its true value in uniformly scaled unit-variance Lorenz63 space.The L2 distance is Euclidean, and corresponding predicted and true unstable steady states are compared in the scaled space.

Double-Scroll Unstable Steady States · Return map · Forecasting task

The paper derives three unstable steady states for the double-scroll system, with the NG-RC reproducing the zero state exactly and closely approximating the positive non-zero state. It evaluates long-term Lorenz63 behavior using return maps and examines how forecasting output weights vary with regularization.

  • Double-Scroll Unstable Steady States: Unstable steady states are obtained by setting the derivatives in Eq. (8) to zero and solving the resulting transcendental equation.The state variables are determined from the steady-state conditions.
  • Double-Scroll Unstable Steady States: Three solutions of the transcendental equation yield three unstable steady states for the parameters used.The three USSs are listed in Eq. (14).
  • Double-Scroll Unstable Steady States: The NG-RC predicts the zero unstable steady state exactly because its odd polynomial powers and absent constant term impose origin symmetry.The L2 distance from the true non-zero positive USS is 2.1±0.2×10-3 in uniformly scaled unit-variance space.
  • Return map: The Lorenz63 z variable’s successive local maxima define a return map that summarizes long-term behavior and enables qualitative system comparisons.The map is constructed by plotting each maximum M_i against the next maximum M_i+1.
  • Return map: Return-map shape depends on the integrator, so both Lorenz63 and double-scroll solutions use an explicit Runge-Kutta 3(2) integrator.The integration method is held consistent across the systems.
  • Return map: The NG-RC’s free-running Lorenz63 forecast is evaluated against Lorenz63 by finding z maxima within a window of 1,000 time units.The same maxima-finding procedure is applied to both systems.
  • Return map: Degree-4 spline interpolation reduces timestep effects and reveals qualitatively good agreement between the NG-RC and Lorenz63 return maps.Interpolated z solutions are used to locate local maxima before constructing Figure 5.
  • Forecasting task: Forecasting output-weight components vary smoothly with regularization parameter α, while many substantial components do not appear directly in the vector field.Figures 6 and 7 identify terms found directly in the vector field with red bars.

Inference Task · Comparing the computational complexity of the NG-RC with a typical traditional RC

The inference task’s output weights are dominated by a constant offset term, while all other components remain non-zero. For forecasting, NG-RC complexity is dominated by ridge regression and is estimated to provide substantial speedup over traditional RC under conservative assumptions.

  • Inference Task: The constant term c is the largest component of W_out because z has a considerable offset.All other W_out components are non-zero and contribute to the final output.
  • Inference Task: The remaining W_out components are non-zero and contribute to the inferred z given x and y.
  • Comparing the computational complexity of the NG-RC with a typical traditional RC: The complexity comparison estimates multiplications and special function evaluations for Lorenz633,9,10 and doublescroll9 forecasting tasks.The NG-RC uses quartic and cubic nonlinear output layers, respectively.
  • Comparing the computational complexity of the NG-RC with a typical traditional RC: NG-RC feature-vector parameters include Nlinear, Nnonlinear, and Ntotal, while traditional RC parameters include N, σr, and Nspecial.The analysis also tracks Mwarmup and Mtrain for warm-up and training steps.
  • Comparing the computational complexity of the NG-RC with a typical traditional RC: The NG-RC’s dominant computational contribution is ridge regression with complexity O((Mtrain(Ntotal)2) over training.For traditional RC, dominant terms include reservoir-state multiplication with the adjacency matrix and ridge regression.
  • Comparing the computational complexity of the NG-RC with a typical traditional RC: The estimated speedup is the summed dominant traditional-RC contributions divided by the corresponding NG-RC sum, indicating substantial speedup even conservatively.Traditional RC implementations trade simulation speed and accuracy differently across References 9, 10, and 3.
  • Comparing the computational complexity of the NG-RC with a typical traditional RC: Around 250 training data points, the NG-RC testing error saturates, indicating that no more data is needed for good generalized performance.The NRMSE is averaged over 20 temporal segments for α=2.5×10-6 in all cases.
  • Comparing the computational complexity of the NG-RC with a typical traditional RC: The required training-set size depends on dt: too-small dt undersamples the attractor, whereas too-large dt may require higher-order nonlinear features.The chosen dt gives about 40 points per Lyapunov time and balances these constraints.

NG-RC performance for the Lorenz63 system driven by noise

The NG-RC was tested on Lorenz63 dynamics driven by substantial Gaussian noise and learned the underlying deterministic system accurately when forecasting noise-free behavior. Good generalization required increasing the ridge parameter.

  • Noise-driven Lorenz63 setup: Gaussian noise with root-mean-square value 1 was added to each Lorenz63 equation, representing about 12% of the variables’ typical magnitudes.The typical root-mean-square values were 7.9, 9.0, and 8.6 for x, y, and z, respectively.
  • NG-RC performance:  = 1.4×10-2 was required for good generalization in the noisy Lorenz63 experiment.The increased ridge parameter was used before forecasting from the final training-data point.
  • NG-RC performance: RMSE = 1.34×10-2 was obtained when NG-RC predictions were compared with noise-free Lorenz63 dynamics, indicating that the model learned the underlying deterministic system.The predictions were evaluated after training by initializing the forecast at the last point in the training data set.
Loading 2106.07688v2…