Source-linked AI summary
Forecasting Chaotic Systems with Very Low Connectivity Reservoir Computers
Aaron Griffith, Andrew Pomerance, Daniel J. Gauthier
TL;DR
Reservoir-computer hyperparameter selection is difficult, especially when common design heuristics have limited applicability. The paper uses Bayesian optimization with a climate-focused performance measure and finds that very low-connectivity reservoirs—including nonrecurrent reservoirs with ρr = 0—can reproduce chaotic-system climates.
Problem
Reservoir hyperparameter spaces are large and difficult to search, while commonly used guidance about spectral radius and recurrence has limited applicability.
Method
The study applies Bayesian optimization to reservoir hyperparameters and evaluates performance using a measure that emphasizes global climate reproduction.
Results
Optimized simpler topologies often perform almost as well as the general-k topology, including topologies with ρr = 0 and no recurrent connections.
Takeaways & Limitations
Reservoirs with very low connectivity, a single cycle, or no cycle can perform as well as more recurrent designs and may simplify software or hardware implementation.
Abstract
from arXiv · showhide
We explore the hyperparameter space of reservoir computers used for forecasting of the chaotic Lorenz '63 attractor with Bayesian optimization. We use a new measure of reservoir performance, designed to emphasize learning the global climate of the forecasted system rather than short-term prediction. We find that optimizing over this measure more quickly excludes reservoirs that fail to reproduce the climate. The results of optimization are surprising: the optimized parameters often specify a reservoir network with very low connectivity. Inspired by this observation, we explore reservoir designs with even simpler structure, and find well-performing reservoirs that have zero spectral radius and no recurrence. These simple reservoirs provide counterexamples to widely used heuristics in the field, and may be useful for hardware implementations of reservoir computers.
I. INTRODUCTION
Reservoir computers use dynamical internal networks for tasks involving time and history, but selecting their hyperparameters is difficult. This study applies Bayesian optimization to climate-focused chaotic-system forecasting and finds effective reservoirs with surprisingly low connectivity.
- Reservoir computers use internal dynamical neural networks that react over time to changing inputs, making them suitable for problems where time and history matter.
- Reservoir construction commonly uses interacting nodes with random or simplified topologies, while task performance is adjusted through large-scale network hyperparameters.
- The hyperparameter space is difficult to search, and common guidance such as spectral radius near unity or recurrent connections has limited applicability.
- Bayesian optimization can handle computationally intensive, noisy discrete or continuous hyperparameter objectives and may identify parameters that defy common reservoir heuristics.
- The study introduces a climate-focused performance measure and finds well-performing reservoirs with extremely low connectivity, including designs with no recurrence and ρr = 0.
A. Reservoir
The reservoir transforms input signals through node dynamics and a trained linear output layer. During forecasting, its output is fed back as input so the system runs autonomously.
- The reservoir dynamics are described by node states whose component-wise nonlinearities determine the internal response.
- The study fixes the reservoir at N = 100 nodes with a three-dimensional input, while Wr and Win encode internal and input-to-node connections.
- The output layer applies a linear transformation to a chosen function of the node values, producing the reservoir computer output.
- Wout is trained by driving the fixed reservoir with known inputs and minimizing the difference between predicted and desired outputs.
- For forecasting, the output replaces the external input, creating an autonomous system whose predictions can be compared with the original signal.
III. EXAMPLE SYSTEMS
The forecasting experiments use three-dimensional chaotic systems with standardized temporal and component scales. For Lorenz ’63, performance is evaluated over one Lyapunov period after training on a preceding trajectory segment.
- The study considers Lorenz ’63, the Rössler system, and a double-scroll chaotic circuit as three-dimensional forecasting inputs.
- The systems are temporally rescaled to share the Lorenz maximum positive Lyapunov exponent λ = 0.9056 and each component is standardized to zero mean and unit variance.
- Lorenz ’63: The Lorenz ’63 attractor is visualized by projecting its three-dimensional trajectory onto the x/z plane.
- Lorenz ’63: The reservoir is trained on one segment of Lorenz dynamics and then predicts the subsequent trajectory using autonomous forecasting.
- Lorenz ’63: Forecasting performance is evaluated over windows of one Lyapunov period, 1/λ = 1.104, because chaotic prediction eventually fails.
B. Rössler
The study includes the Rössler system and a double-scroll chaotic circuit as additional examples. A logarithmic transformation is used for the Rössler z component because it otherwise remains near zero with rare positive spikes.
- Rössler: The Rössler system is one of the chaotic systems used as a forecasting example.
- Rössler: The Rössler z component is transformed to logz for both reservoir input and prediction output because it mostly stays near zero with rare positive spikes.
- Double-scroll circuit: The double-scroll chaotic circuit is described using dimensionless equations.
- Double-scroll circuit: The double-scroll circuit uses parameters R1 = 1.2, R2 = 3.44, R4 = 0.193, Ir = 2.25× 10−5, and α = 11.6.
IV. RESERVOIR CONSTRUCTION AND TRAINING
The reservoir computer is constructed by building an internal reservoir, connecting nodes to the input, and training the output weights.
- Construction combines an internal reservoir, input connections, and training of Wout.
A. Internal Reservoir Construction
The study constructs reservoirs by controlling input connectivity, recurrent in-degree, spectral radius, and characteristic time scale, then compares increasingly simple network topologies. Bayesian optimization found that k = 1 reservoirs could perform as well as higher-connectivity networks, motivating tests of cut cycles, simple cycles, and delay lines.
- Hyperparameters: Input connections are randomized using σ, while reservoir structure is controlled by k and ρr.σ determines the probability that a node connects to a reservoir input; k is recurrent in-degree; ρr is spectral radius.
- Hyperparameters: The construction searches five hyperparameters, including γ for characteristic time scale, using Bayesian optimization to minimize forecasting error.
- Topology constraints: k = 1 networks often contain disconnected components, so the study limits analysis to reservoirs with a single connected component.Disconnected components effectively act as RCs operating in parallel and are not considered further.
- Topology constraints: A connected k = 1 network contains one directed cycle with directed trees branching from it; cutting the cycle produces a tree.
- Topologies: The comparison includes general fixed-in-degree, connected k = 1, cut-cycle, simple-cycle, and delay-line reservoirs.Simple-cycle reservoirs contain only a cycle, whereas delay-line reservoirs contain a single line of nodes.
- Topologies: After cutting the cycle, both cut-cycle and line reservoirs have ρr = 0 despite being rescaled to a fixed ρr beforehand.
B. Bayesian Optimization
The study uses Bayesian optimization because reservoir hyperparameters include discrete and noisy quantities that challenge grid search and gradient descent. The optimizer repeatedly tests randomly instantiated reservoirs across several topologies and updates the hyperparameters based on measured forecasting performance.
- Grid search and gradient descent struggle with non-continuous parameters, noisy results, or both.
- Bayesian optimization handles noise and integer parameters such as k, is more efficient than grid search, and requires minimal tuning.
- The searched ranges include values suggested by existing heuristics while permitting broader exploration within a runtime constraint.The authors identify expanding these ranges as valuable future exploration, especially for k.
- At each iteration, the optimizer selects hyperparameters, constructs and trains one random reservoir, measures ε, and chooses a subsequent test point.
- Figure 3 compares five internal topologies, including fixed-in-degree, connected k = 1, cut-cycle, simple-cycle, and delay-line networks.The figure omits connections to the reservoir input and output layers.
C. Training
Training integrates the reservoir with the chosen input over transient, training, and testing intervals, then fits Wout by regularized regression. The procedure also breaks a symmetry that can confuse prediction when the reservoir is combined with the Lorenz system.
- Integration and data splits: The reservoir is integrated with the chosen input from t = 0 to 300 using Δt = 0.01.
- Integration and data splits: The testing period runs from t = 200–300, after an earlier transient and training period.The transient reduces dependence on specific initial conditions; training data are used only for fitting and testing data only for evaluation.
- Symmetry handling: Before integration, the procedure breaks a reservoir–Lorenz symmetry that can confuse prediction.
- Output training: The output weights Wout are fitted after applying the same symmetry-breaking procedure across the three input systems.
- Output training: Ridge regression selects the regularization parameter α from 10^-5 to 10^5 by leave-one-out cross-validation.Optimizing α separately avoids adding a Bayesian-optimization dimension and does not require re-integration.
V. FORECASTING AND EVALUATION
The usual single-start forecasting error can favor reservoirs that fail to reproduce the Lorenz climate. Evaluating forecasts from multiple testing-period starting points instead emphasizes global attractor reproduction and improves Bayesian optimization.
- Single-start evaluation: The standard evaluation initializes forecasting at t1, compares one Lyapunov-time forecast with the true system, and computes RMSE ε1.The input signal is normalized to unit variance, so ε1 is normalized.
- Single-start evaluation: Low ε1 can be misleading because reservoirs may forecast well near t1 while failing to reproduce the Lorenz attractor.One example produces a lower ε1 than every optimized reservoir despite visibly failing to learn the attractor.
- Single-start evaluation: Optimizing ε1 can waste search effort on reservoirs that perform well only near t1 and poorly reproduce the climate elsewhere.The choice of t1 can also strongly affect ε1 near the Lorenz system’s unstable saddle point.
- Averaged evaluation: The alternative evaluation uses 50 evenly spaced testing-period starts, producing 50 short-term errors from data not used to construct Wout.The starts lie between t = 200 and 300, and each forecast is evaluated as in the standard procedure.
- Averaged evaluation: Combining errors across attractor locations yields a figure of merit that measures average forecasting ability and emphasizes global dynamics rather than one temporal segment.This averaged ε avoids optimization valleys with low ε1 that fail to reproduce the attractor.
VI. RESULTS
Bayesian optimization found that simpler reservoir topologies can match the performance of general reservoirs, including designs with zero spectral radius and no recurrent connections. Performance distributions still differ substantially, while optimized reservoirs showed mixed transferability across dynamical systems.
- Optimization results: After 100 Bayesian-optimization iterations, simpler topologies performed nearly as well as the general topology.The study recorded the best-performing reservoir for each topology and estimated variability across 20 optimization repeats.
- Optimization results: Topologies (c) and (e) achieved comparable performance despite having zero spectral radius and no recurrent connections.The remaining topologies had spectral radius much less than 1, challenging the usual expectation of spectral radius near 1.
- Performance distributions: For ε < 0.3, reservoirs consistently reproduced the Lorenz attractor; for 0.3 < ε < 1.0, they reproduced part of it before failing; above ε > 1.0, failures were catastrophic.These qualitative regions were identified by evaluating 200 reservoirs of each topology with optimized hyperparameters.
- Performance distributions: Topology (a) was more likely than a single-cycle topology (d) to produce a well-performing reservoir, while its performance was similar to tree-like topology (c).Thus, best-case performance and the probability of obtaining it differed across topologies.
- Implementation implications: Simpler topologies may benefit hardware implementations despite their wider performance distributions, while their structured weight matrices can also accelerate software integration.The authors note that automated testing can help hardware designs evaluate many candidate reservoirs.
- Other systems: All five topologies performed similarly after optimization on the Rössler and double-scroll systems, and optimization almost always succeeded within 100 iterations.These results extended the Lorenz findings to two additional chaotic systems.
- Cross-system reuse: Lorenz-optimized reservoirs generally performed poorly when reused for double-scroll prediction, although some best performers approached reservoirs optimized specifically for double-scroll.The results indicate that some reservoirs can work across systems, but transfer performance varied greatly.
- Lorenz attractor reproduction: All five optimized reservoirs reproduced the Lorenz attractor well in 100-time-unit free-running predictions.The attractor plots were treated as qualitative checks rather than quantitative comparisons.
VII. CONCLUSION
Bayesian optimization identifies effective reservoir computers using global-climate evaluation and reveals very low-connectivity designs, including reservoirs without recurrence. These designs challenge common heuristics, but their performance is established only in narrow software-based chaotic forecasting contexts.
- VII. CONCLUSION: Bayesian optimization can quickly identify high-performance reservoirs when evaluation averages performance across many points on the attractor.This steers optimization away from reservoirs that forecast well only near one initial point.
- VII. CONCLUSION: Optimized reservoirs often have extremely low connectivity, including designs with a single internal cycle or no cycle at all.Some well-performing reservoirs have no recurrent connections and spectral radius ρr = 0.
- VII. CONCLUSION: These simple topologies challenge widely used reservoir-design heuristics and produce simpler weight matrices that may benefit software integration and hardware implementation.Hardware benefits may arise when node connections have a cost or recurrence is difficult to implement.
- VII. CONCLUSION: The best low-connectivity reservoirs perform as well as more complicated reservoirs, although they perform worse on average.Finding the best instance may still require only a few trials.
- VII. CONCLUSION: The results are limited to software-based chaotic-system forecasting, while generalization to classification, hardware reservoirs, and simultaneous multi-task performance remains future work.The authors also propose further study of attractor-overlap metrics and possible explanations for similar best-performing reservoirs across topologies.