Chaos as an interpretable benchmark for forecasting and data-driven modelling

1 Introduction

• Chaotic systems generate deterministic but non-repeating measurements, allowing attractor structure to be resolved in ever-increasing detail.Their trajectories do not recur or intersect, despite the systems having finite representations.Source: §1 · p. 1
• Their mathematical structure and accessible generating processes make chaotic systems controllable benchmarks that can connect model performance to system properties.The authors identify this connection as a way to aid interpretation of complex models.Source: §1 · p. 2
• Prior use of individual systems such as the Lorenz attractor motivates a broader database, since isolated examples limit standardization and systematic comparison across diverse dynamics.The authors note that emphasis on single systems may undermine generalizability and prevent systematic interpretation of algorithm performance relative to system properties.Source: §1 · pp. 1–2
• The database pairs published low-dimensional chaotic systems with time series, mathematical annotations, and explicit forms that can be re-integrated for new data.The authors also benchmark forecasting and demonstrate transfer learning for classification, importance sampling, and symbolic regression applications.Source: §1 · p. 2

2 Description of Datasets

• The dataset focuses on low-dimensional strange attractors with bounded, stationary, ergodic dynamics, excluding transient chaos, chaotic repellers, and most nonchaotic strange attractors.Chaos is defined as exponential divergence between copies initialized in infinitesimally different states; one quasiperiodic two-dimensional torus is retained as an exception.Source: §2 · p. 3
• The collection currently includes 131 named chaotic systems, with implementations, citations, initial conditions, precomputed trajectories, and regression data with time derivatives.Records include train and test trajectories at coarse and fine granularities, in multivariate and univariate forms, with and without Brownian noise.Source: §2 · p. 3
• Annotations include estimates of standard mathematical characteristics and qualitative properties such as Hamiltonian or dissipative behavior, boundedness, and time dependence.The collection also includes Lyapunov-spectrum-derived quantities, including the Pesin entropy bound and Kaplan–Yorke fractal dimension.Source: §2 · p. 3
• Phase-surrogate testing selects integration and sampling rates to align dominant and minimum significant timescales across systems.The authors also test ergodicity for consistent initial conditions and use continuous re-orthonormalization when computing mathematical quantities such as Lyapunov exponents.Source: §2 · p. 4
• Embedding and clustering based on 787 time-series features separates systems into groups such as smooth scroll-like and spiking dynamics, with a filamentary manifold in embedding space.The authors relate the manifold to the rarity of chaotic attractors and their occurrence between bifurcations to simpler dynamics.Source: §2 · p. 4

2.1 Prior Work.

• Dataset size is flexible: each system can be re-integrated at arbitrary lengths and sampled at any granularity, with new initial conditions, altered stochastic forcing, or perturbed parameters.The work systematizes prior studies benchmarking methods on single chaotic systems; expansion to new systems requires manually reproducing reported dynamics and identifying suitable parameters and initial conditions.Source: §2 · p. 3, §2.1 · p. 5

Forecasting

• NBEATS and Transformer achieve the lowest median forecasting scores, with NBEATS strongest across varied systems and granularities and remaining effective under noise.Model rankings remain stable across two orders of magnitude in sampling granularity; under noise, performance converges as overall scores decline.Source: Task 1: Forecasting · p. 6
• Forecast error correlates strongly with the largest Lyapunov exponent across systems, while correlations with attractor dimension and entropy are weaker.The authors describe this as the first large-scale test of the empirical relevance of Lyapunov exponents for forecasting across several periods.Source: Task 1: Forecasting · pp. 5–6

Task 3: Transfer learning and data augmentation

• Task-matched surrogate pretraining raises average UCR classification accuracy by 4% ± 1% over raw data and 2% ± 1% over random-timescale surrogates.Both comparisons are statistically significant (p < 10−4); the random-timescale control isolates the contribution of matching timescales.Source: Table 1 · p. 7, Task 3: Transfer learning and data… · p. 8

Task 2: Accelerating model training with importance sampling

• Importance sampling achieves significantly lower forecast error than full-set and unweighted-sampling controls while requiring substantially less computation.Training samples are selected in proportion to historical forecast error, and perturbed attractor points generate trajectories for subsequent training.Source: Task 2: Accelerating model training… · p. 7

Task 3: Transfer learning and data augmentation

• On EOGHorizontalSignal, classification accuracy scales monotonically as the number of systems used for pretraining increases.The figure reports bootstrapped standard errors, sampling dynamical systems with replacement.Source: Task 3: Transfer learning and data… · p. 8, Figure 3 · p. 8

Task 4: Data-driven model inference and symbolic regression

• PySR and DSR have nearly equivalent low symbolic-regression errors, while polynomial PySINDY is fast at 0.01 ± 0.01 s per system versus 1400 ± 60 s for PySR and 4300 ± 200 s for DSR.The authors attribute strong results partly to the relatively simple algebraic construction of many named systems in the collection.Source: Task 4: Data-driven model inference… · p. 9
• Symbolic-regression rankings depend on equation complexity, operator libraries, data, compute, and nonlinearity; the experiments also omit measurement noise.The authors note that noisier settings may favor DSR and that harder benchmarks could include nested expressions and uncommon transcendental functions.Source: Task 4: Data-driven model inference… · p. 10
• DSR errors increase consistently with the degree of chaos, a pattern the authors hypothesize may reflect a broader search space of local partial formulae.Figure 4 compares test-error distributions across systems and correlates errors with mathematical properties.Source: Figure 4 · p. 9, Task 4: Data-driven model inference… · p. 10

4 Discussion

• The collection systematizes testing on single chaotic systems and supports customizable time-series benchmarks and applications such as transfer learning.The authors identify feedback-based control and numerical-propagator inference as potential applications not explored here.Source: §4 · p. 10
• Lyapunov exponent correlates with empirical predictability across forecasting models, while more chaotic systems are harder to model with symbolic regression.The authors attribute the symbolic-regression pattern to the diverse local approximants available for complex dynamical systems.Source: §4 · p. 10
• The collection includes only previously published chaotic systems, requiring manual curation and implementation and potentially biasing it toward unusual dynamics.Scoring primarily measures point-wise forecast accuracy; long-timescale dynamical properties and predicted-attractor topology may be informative alternatives.Source: §4 · p. 10

Supplementary material

• The passage identifies an arXiv record as arXiv:2110.05266v2, places it in cs.LG, and displays the date 29 Jan 2023.Source: data-driven modelling” · p. 17

I. DATA AVAILABILITY

• The repository provides benchmark code and precomputed datasets under an Apache 2.0 license, while the included equations are public domain and require no license.The benchmark subdirectory contains code to reproduce the benchmarks, figures, and tables.Source: §I · p. 17

II. DESCRIPTIONS OF ALL SYSTEMS

• The systems are documented with descriptions and citations, visualized in Figure S1, and represented in Table S2 by recorded properties.Repository entries contain full records and descriptions for each system.Source: §II · pp. 19–20, Table S2 · p. 22

III. DATASET STRUCTURE AND FORMAT

• Python objects expose system properties and equation implementations, load or re-integrate trajectories, and support bulk loading or re-integration.Each system includes 16 precomputed series spanning sampling granularity, splits, univariate or multivariate views, and noisy or unforced trajectories.Source: §III · p. 22

IV. GLOSSARY

• The glossary defines terms including attractor, bifurcation, and ergodicity, with ergodicity connecting long runs to forecasting on shorter trajectories from different initial points.It describes a strange attractor as bounded, nonperiodic motion with local instability and fractal appearance.Source: §IV · pp. 22–23

V. CALCULATION OF MATHEMATICAL PROPERTIES

• Timescales are aligned using power spectra and random-phase surrogates; mathematical properties are estimated from replicated, high-granularity trajectories.Lyapunov exponents use vector evolution and Gram–Schmidt re-orthonormalization, while multiscale entropy is calculated per variable and its median recorded, with expected overestimation because cross-variable motifs are omitted.Source: §V · pp. 23–24

VI. STATISTICAL FEATURES AND EMBEDDING

• The feature analysis computes 787 time-series features, retains 100 high-variance features, embeds systems with UMAP, and identifies eight clusters with affinity propagation.The reported embedding for each system is the median across 40 random-initial-condition replicates.Source: §VI · p. 24

Models

• The benchmark compares deep-learning, statistical, machine-learning, classical forecasting, and naive baseline models, with untuned settings kept at reference-library defaults.The model set includes NBEATS, Transformer, LSTM, Temporal Convolutional Network, Prophet, ARIMA, Random Forest, and naive mean, seasonal, and drift baselines.Source: §VII · p. 24

Hyperparameter tuning

• Hyperparameters are grid-searched separately for each model, system, and granularity using 10 training periods and 2 validation periods.Coarse and fine datasets use different search ranges, including one timestep, half a period, and one full period.Source: §VII · pp. 24–25

Scoring

• Testing uses a trajectory from another initial condition on the same attractor, training on its first 10 periods and scoring the remaining 2 with nine forecast metrics.Recorded metrics include MAPE, SMAPE, MAE, MSE, RMSE, r2, and MASE, among others.Source: §VII · p. 25

• Median forecasting performance degrades for all methods in the noisy experiments.Source: A. The effect of noise on forecasting… · p. 25
• Noise weakly affects naive baselines, while deep learning models continue to perform very well.Source: A. The effect of noise on forecasting… · p. 25
• Random forest performance noticeably degrades, whereas Fourier regression outperforms several more sophisticated models with noise.The authors hypothesize that high-frequency noise obfuscates phase information, making periodic-basis methods such as Fourier and N-BEATS less affected.Source: A. The effect of noise on forecasting… · p. 25

VIII. FORECASTING EXPERIMENTS AS GRANULARITY AND NOISE ARE VARIED

• Granularity is points sampled per period, based on the dominant significant frequency; noise level 0.2 means noise amplitude is 20% of signal standard deviation.Timescale-unit hyperparameters, such as time lags and input chunk size, are rescaled for the granularity sweep.Source: §VIII · p. 25
• As noise exceeds the signal’s average amplitude, model performance converges because less usable signal information remains for forecasting.Relative model rankings remain somewhat stable as noise increases.Source: §VIII · p. 25
• Deep learning models, particularly NBEATS, perform well across granularities, but their errors fluctuate systematically with granularity.The authors suggest default architectures may have timescale bias and caution that exhaustive, computationally expensive hyperparameter tuning is needed to investigate it.Source: §VIII · p. 25
• Figure S2 compares forecast-error distributions across models and coarse versus fine sampling, with separate panels for noise-free and noisy results.Model order differs between panels because relative model performance changes with noise.Source: Figure S2 · p. 26

IX. RELATIVE PERFORMANCE OF FORECASTING MODELS ACROSS DIFFERENT MATHEMATICAL PROPERTIES

• The mathematical-property analysis uses sliding 20% quantile windows for properties including Lyapunov exponent, correlation dimension, and multiscale entropy, then computes model errors within each window.Figure S4 plots median results across systems in each quantile window.Source: §IX · p. 26, Figure S4 · p. 27

• A single common hyperparameter set is used for the LSTM across systems, so its baseline error is higher than in benchmarks tuned separately for each system.Source: §X · p. 27
• The LSTM importance-sampling procedure selects timepoints in proportion to historical backtest error, perturbs their attractor locations, and simulates new training trajectories.It uses trajectories of τ = 150 timesteps, b = 30 training epochs per update, and ν = 5 meta-epochs.Source: §X · p. 27
• Training alternates between backtesting the training trajectory and using newly simulated trajectories for subsequent training epochs.Source: §X · p. 27
• The random-batch control repeats the procedure with randomly sampled timepoints, while the full-epoch baseline uses standard training.The random-batch control addresses the possibility that performance improves simply with more training data.Source: §X · p. 27

XI. TRANSFER LEARNING EXPERIMENTS

• The transfer-learning experiments retain 96 of 128 UCR classification tasks, excluding datasets with fewer than 100 valid timepoints.Source: §XI · p. 27
• For each task, the method estimates its dominant frequency, matches trajectory granularity, and trains an encoder on re-integrated trajectories from all 131 chaotic systems.A random-timescale ablation instead chooses unrelated granularities.Source: §XI · p. 27
• Encoded task data are converted to 787 tsfresh features and passed to a cross-validated ridge classifier; the no-transfer baseline uses the original task data.Test accuracy is reported, with timescale extraction, surrogate generation, and model fitting restricted to training data.Source: §XI · p. 28

XII. SYMBOLIC REGRESSION EXPERIMENTS

• The symbolic-regression benchmark uses trajectory inputs and the governing differential equation’s right-hand side as targets, with 150 points per train and test trajectory.Each trajectory spans 10 dominant-timescale periods at 15 points per period; the authors note target-formula accuracy may be reduced in low-measure attractor regions.Source: §XII · p. 28
• Symbolic-regression expressions are fitted separately for each dynamical variable and evaluated as forecasts on an unseen test trajectory using the forecasting error metrics.The overall system error is the median across variables.Source: §XII · p. 28

XIII. NEURAL ORDINARY DIFFERENTIAL EQUATION EXPERIMENTS

• For each system, an nODE was trained on 1,000 multivariate timepoints and evaluated on a 200-timepoint trajectory from an unseen initial condition using sMAPE.Predicted trajectories were compared with trajectories generated by the true system.Source: §XIII · p. 28
• nODE forecasting was competitive with other time-series techniques, but trajectories often became trapped near unstable periodic orbits over long durations.The authors suggest shadowing events in training data dominate the learned representation.Source: §XIII · p. 28
• The largest observed Spearman correlation between nODE quality and a mathematical property was 0.072 ± 0.003, not significantly different from zero.The value is the median, with standard error determined by bootstrapping.Source: §XIII · p. 28

1. Motivation

• The dataset is intended as a generative benchmark for time-series mining, producing arbitrary synthetic data through deterministic processes.The authors note a possible privacy concern if improved time-series models identify latent factors that could deanonymize physiological recordings.Source: §1 · p. 29

2. Composition

• Each instance includes system equations, parameters, initial conditions, timescales, mathematical properties, and 16 precomputed trajectories; the analyses cover 131 continuous-time systems, while excluding 30 discrete maps.The collection is a sample of previously published systems, not the full space of possible chaotic systems.Source: §2 · p. 29

3. Collection

• Systems were selected from published work with explicit equations, parameters, and chaotic initial conditions, then reimplemented and checked against reported dynamics.The collection process also determined sampling timescales and calculated properties such as Lyapunov exponents and entropy; unavailable references or context are left blank, but every system can be integrated.Source: §3 · p. 30
• Because the collection samples named and published systems, it is not representative of all low-dimensional chaotic systems and should not estimate their prevalence or common motifs.It is representative of chaotic dynamics as they appear in the literature.Source: §3 · p. 30

4. Preprocessing

• Phase-surrogate testing identifies significant frequencies to set integration timesteps and sampling rates, making dynamical variation comparable across systems while resolving their dynamics.Source: §4 · p. 30

5. Distribution

• The dataset is distributed on GitHub under an Apache 2.0 license, with no fees; the authors retain repository control and responsibility for rights violations.They state that they will ensure continued repository accessibility.Source: §5 · p. 31, §XV · p. 31

6. Legal

• The dataset contains no individual or protected-subject data, and the authors state that no institutional approval or consent is required.They report, to their knowledge, that it contains no personal information and complies with GDPR and equivalent standards.Source: §6 · p. 31