PaperlayerView exact revision on arXiv

Source-linked AI summary

Reservoir computing approaches for representation and classification of multivariate time series

Filippo Maria Bianchi, Simone Scardapane, Sigurd Løkse, Robert Jenssen

arXiv:1803.07870v3cs.NE

TL;DR

MTS classification needs accurate representations, while standard RC methods provide speed but can lag fully trainable neural networks. The paper introduces reservoir model space and a modular RC framework, finding faster classification and superior accuracy with the proposed representation across benchmark and real-world datasets.

  • Problem

    Standard RC classifiers train extremely quickly but fail to achieve the accuracy of fully trainable neural networks for MTS classification.

  • Method

    The paper learns an unsupervised reservoir model-space representation from linear predictors of low-dimensional reservoir dynamics and provides a modular RC framework for classification.

  • Results

    Across benchmark and real-world MTS datasets, RC classifiers are dramatically faster and achieve superior classification accuracy when using the proposed representation.

  • Takeaways & Limitations

    The reservoir model space provides an informative MTS representation whose combination with more sophisticated architectures improves accuracy only slightly.

Abstract

from arXiv · show

Classification of multivariate time series (MTS) has been tackled with a large variety of methodologies and applied to a wide range of scenarios. Reservoir Computing (RC) provides efficient tools to generate a vectorial, fixed-size representation of the MTS that can be further processed by standard classifiers. Despite their unrivaled training speed, MTS classifiers based on a standard RC architecture fail to achieve the same accuracy of fully trainable neural networks. In this paper we introduce the reservoir model space, an unsupervised approach based on RC to learn vectorial representations of MTS. Each MTS is encoded within the parameters of a linear model trained to predict a low-dimensional embedding of the reservoir dynamics. Compared to other RC methods, our model space yields better representations and attains comparable computational performance, thanks to an intermediate dimensionality reduction procedure. As a second contribution we propose a modular RC framework for MTS classification, with an associated open-source Python library. The framework provides different modules to seamlessly implement advanced RC architectures. The architectures are compared to other MTS classifiers, including deep learning models and time series kernels. Results obtained on benchmark and real-world MTS datasets show that RC classifiers are dramatically faster and, when implemented using our proposed representation, also achieve superior classification accuracy.

I. INTRODUCTION

Multivariate time-series classification assigns sequences to fixed classes across applications, while recurrent approaches process inputs dynamically and classify their generated states. The paper addresses limitations of standard reservoir representations with a model-space representation and a modular framework for efficient classification.

  • I. INTRODUCTION: MTS classification assigns each multivariate time series to one of a fixed number of classes across application domains.Examples include health monitoring, civil engineering, action recognition, and speech analysis.
  • I. INTRODUCTION: Standard RC classifiers summarize reservoir states using the last or mean state, but their training-speed advantage has not matched competing classifiers’ accuracy.Model-based alternatives instead represent an MTS through parameters trained to predict future inputs from reservoir states.
  • I. INTRODUCTION: The reservoir model space represents each MTS with parameters of a one-step-ahead predictor for future reservoir states rather than future inputs.The representation is intended to characterize the dynamics captured by the reservoir.
  • I. INTRODUCTION: The proposed framework combines reservoir, dimensionality-reduction, representation, and readout modules to implement multiple RC architectures.It includes an associated open-source Python library and supports independently selected architectural components.
  • I. INTRODUCTION: Experiments compare the framework with fully trainable RNNs, deep learning models, time-series classifiers, DTW, and kernel-based SVMs.The reported results show RC classifiers are dramatically faster and achieve competitive accuracy with the proposed representation.

A. Fully trainable RNNs and gated architectures

Fully trainable RNN classifiers jointly learn encoder and decoder parameters by minimizing an empirical loss, using back-propagation through time. Training difficulty motivates alternatives such as RC, LSTM, and GRU architectures with different complexity and memory trade-offs.

  • A. Fully trainable RNNs and gated architectures: Fully trainable RNNs jointly learn encoder and decoder parameters by minimizing an empirical cost over the training MTS.Gradients with respect to both parameter sets are computed through back-propagation through time.
  • A. Fully trainable RNNs and gated architectures: The training objective uses a generic loss function, such as cross-entropy over class labels.The encoder and decoder parameters are optimized together under this empirical cost.
  • A. Fully trainable RNNs and gated architectures: Dropout and ℓ2 penalties can regularize fully trainable recurrent classifiers during optimization.The experiments use dropout specialized for recurrent architectures.
  • A. Fully trainable RNNs and gated architectures: RNN training is hindered in practice by gradient shrinkage during temporal back-propagation, despite basic RNNs’ theoretical dynamical-system capability.Stability constraints on recurrent derivatives contribute to this training difficulty.
  • A. Fully trainable RNNs and gated architectures: LSTM gating preserves internal memory over long intervals but increases computational and architectural complexity.GRUs are presented as a lower-parameter alternative with better memory conservation than LSTMs.

B. Reservoir computing and output model space

Reservoir computing avoids back-propagation by fixing randomly generated recurrent parameters and learning a readout from reservoir dynamics. Output model space instead encodes each MTS in parameters of a one-step predictor, providing a richer representation than basic state summaries.

  • B. Reservoir computing and output model space: RC randomly generates and then fixes encoder parameters, avoiding the costly operation of back-propagating through time.A large reservoir supplies heterogeneous dynamics, with behavior controlled by spectral radius, connection sparsity, hidden-unit count, and input scaling.
  • B. Reservoir computing and output model space: In ESNs, the decoder or readout is commonly a linear model whose parameters can be learned through a closed-form ridge-regression solution.An untrained reservoir combined with a linear readout defines the basic ESN model.
  • B. Reservoir computing and output model space: Output model space represents an MTS using parameters of a ridge model trained to predict the input one step ahead from reservoir states.Those parameters are then processed by a classifier.
  • B. Reservoir computing and output model space: The output model-space representation depends on the entire reservoir-state sequence rather than only its final state.The paper introduces a new model space to obtain a more expressive input representation.

III. PROPOSED RESERVOIR MODEL SPACE REPRESENTATION

The reservoir model space represents each multivariate time series through a linear predictor of its next reservoir state, after dimensionality reduction. This representation characterizes both the input and the generative process of its high-level dynamical features.

  • The paper introduces reservoir model space as an unsupervised representation of a multivariate time series based on predicting future reservoir states in a low-dimensional latent space.The approach extends prior representation methods by modeling reservoir dynamics rather than directly predicting future inputs.
  • The predictor accounts for all reservoir dynamics instead of selecting only dynamics useful for a fixed prediction horizon.This design addresses the bias introduced when task-specific prediction discards dynamics that may characterize the time series.
  • Each time series is encoded by the parameters θh = [vec(Uh); uh] of its linear model for predicting the next reservoir state.The parameter vector is the representation supplied to downstream classifiers.
  • The reservoir model representation characterizes both the input and the generative process of its high-level dynamical features and induces metric relationships between samples.A classifier using this representation combines the explanatory capability of generative models with discriminative classification.

B. Dimensionality reduction for reservoir states tensor

The dimensionality-reduction module makes reservoir model-space representations tractable by reducing reservoir features while preserving temporal structure across each time series. The proposed tensor treatment avoids mixing samples and time steps during covariance estimation.

  • High-dimensional reservoir predictors can make the representation intractable through overfitting and costly ridge-regression solutions for each time series.Dimensionality reduction is therefore needed before constructing the model-space representation.
  • The reservoir activations are represented as a 3-mode tensor H ∈ R^N×T×R, with reduction mapping the feature dimension R to D where D ≪ R.The sample and temporal dimensions remain unchanged during the transformation.
  • Standard PCA on concatenated samples and time steps mixes those dimensions, potentially losing within-sample temporal variation.The resulting covariance captures variation across samples and time steps simultaneously, ignoring temporal modes within individual time courses.
  • The proposed covariance treats each Hn ∈ R^T×R as one observation, preserving temporal ordering across multivariate time series.The reduced tensor is obtained with the first D eigenvectors through the 3-mode product Ĥ = H ×3 E.
  • After reduction, each representation uses parameters θh = [vec(Uh); uh] ∈ R^D(D+1), replacing the unreduced feature dimension R.The procedure reduces the parameter count while retaining the model-space construction.
  • Computing all representations costs O(NTVH) for reservoir states plus O(H^2NT + H^3) for dimensionality reduction.

IV. A UNIFIED RESERVOIR COMPUTING FRAMEWORK FOR TIME SERIES CLASSIFICATION

The paper introduces a unified reservoir-computing framework that combines alternative reservoir, reduction, representation, and readout modules. It supports standard and advanced architectures, including bidirectional reservoirs and the proposed model-space representation.

  • The framework generalizes several reservoir-computing architectures by combining reservoir, dimensionality-reduction, representation, and readout modules.Models are formed by selecting one variant from each module.
  • A. Reservoir module: The reservoir processes each input multivariate time series into a state sequence and can be either unidirectional or bidirectional.Bidirectional processing feeds the same reservoir the input in forward and reverse order.
  • Representation module: The framework includes the reservoir model space as a representation option alongside other reservoir-based encodings.
  • A. Reservoir module: A bidirectional reservoir concatenates forward and reverse state vectors to summarize recent and past information at each step.The resulting representation can capture longer temporal dependencies.
  • Representation module: Bidirectional model-space prediction jointly targets the next state and the previous state, or their low-dimensional embeddings.This combines prediction and memorization objectives across both temporal directions.
  • Readout module: The architecture trains a decoder to predict labels from the encoder-generated representation, with mutually exclusive module choices defining different models.An open-source Python library implements the unified framework.

B. Dimensionality reduction module

The framework’s dimensionality-reduction module offers unsupervised projections such as PCA and kernel PCA, but reduction can be skipped. It is especially important for model-space and bidirectional configurations whose representations grow larger.

  • The dimensionality-reduction module projects reservoir activations onto a lower-dimensional subspace using unsupervised criteria.PCA and kernel PCA project onto the first D covariance eigenvectors.
  • Dimensionality reduction is optional, but particularly important for reservoir model-space and bidirectional reservoirs because their representations are larger.
  • Readout module: The readout module classifies representations using a linear model, SVM, or MLP.A linear readout is fast to train, while nonlinear readouts can provide greater representational power.
  • Experimental setup: The experiments compare RC architectures with trainable LSTM and GRU classifiers, DTW nearest-neighbor, kernel SVMs, deep learning models, and other time-series methods.
  • Datasets: The evaluation uses datasets from the UCR, UEA, and UCI repositories, plus univariate datasets and a blood-sample infection-classification case study.The blood dataset contains 10 measurements collected over 20 days after surgery for each patient.
  • Experimental setup: The proposed rmESN achieves competitive results with fixed hyperparameters, indicating robustness and a practical advantage over traditional RC approaches.

A. Performance comparison on benchmark datasets

On benchmark datasets, reservoir model-space representations substantially improve RC classification accuracy while preserving RC’s major speed advantage. The proposed rmESN combines the strongest overall accuracy with execution times comparable to basic lESN.

  • rmESN achieves the best overall mean accuracy among the compared classifiers, with low standard deviation across reservoir initializations.Its model-space representation produces consistently good results despite randomized reservoir initialization.
  • Model-space RC classifiers outperform basic lESN, while LSTM and GRU improve on lESN but remain below the model-space RC result.The comparison includes fully trainable RNNs and 1-NN with DTW.
  • RC classifiers require only a few seconds on average for training and testing, whereas LSTM and GRU require more than 20 minutes on average.1-NN with DTW is also much slower than the RC methods.
  • Dimensionality reduction makes rmESN executable in a time comparable to lESN while retaining the stronger model-space representation.This procedure is identified as the reason rmESN avoids the additional computational burden of its representation.

B. Experiments with bidirectional reservoir and deep-readout

Bidirectional reservoirs and deep readouts improve RC classification accuracy, with the largest gains occurring for weaker representations. Across the broader ranking and blood-sample experiment, model-space representations remain central to accuracy, while added readout complexity trades accuracy or robustness against computation.

  • Bidirectional reservoir: Bidirectional reservoirs improve every RC classifier, with the largest improvement occurring for lESN; bi-rmESN reaches the highest overall accuracy.The gain is attributed to capturing early-sequence information and dependencies beyond a unidirectional reservoir’s memory.
  • Deep readout: Deep readouts improve lESN more than omESN or rmESN, while adaptive activations increase modeling resources and can improve accuracy and robustness.Maxout slightly increases training time but improves average accuracy and reduces accuracy standard deviation.
  • Overall ranking: The proposed reservoir model-space representation is the key factor associated with the highest mean-accuracy ranking across the evaluated MTS classifiers.Further additions through deep readouts and bidirectional reservoirs improve performance beyond the representation alone.
  • Blood-sample MTS: On blood-sample MTS, svm-rmESN slightly outperforms SVM classifiers using LPS and TCK kernels, while remaining much faster than those kernels.The SVM readout increases RC training time slightly, but RC methods retain a substantial speed advantage.
  • Blood-sample MTS: The SVM readout improves rmESN only slightly but produces a much larger accuracy increment for omESN.Standard deviations are high because random data splits change the classification task at each iteration.

VI. COMPARISON WITH DEEP LEARNING BASELINES ON THE CLASSIFICATION OF UNIVARIATE TIME SERIES

The proposed RC approach is evaluated on univariate time-series classification, where stronger task-specific biases can support high performance. Across these tasks, it remains competitive while retaining the framework’s efficiency and flexible model-space design.

  • Experimental setting: Univariate datasets were added as a supplementary evaluation because task-specific algorithms can exploit stronger biases than in multivariate classification.The experiments used datasets from timeseriesclassification.com and regular PCA instead of the proposed multivariate temporal-data extension.
  • Baselines: The comparison included LSTM-FCN, MMCL, and TSML alongside the RC classifiers.
  • Results: The rmESN combines a simple linear readout with a reservoir model-space representation, providing a competitive alternative to more complex models.
  • Results: The RC classifiers were reported to achieve superior performance in accuracy and execution time on the considered real-world datasets.
  • Results: On univariate time-series tasks, the proposed approach achieved competitive results compared with state-of-the-art methods.

APPENDIX

The appendix examines how reduced model-space dimensionality affects training time and accuracy, selects D = 75, and describes ranking and significance analyses for benchmark classifiers.

  • Dimension selection: Average training time and classification accuracy were evaluated across different reduced-space dimensions using k-fold cross-validation.The averages are reported in Fig. 7.
  • Dimension selection: D = 75 is selected because classification accuracy shows an elbow there, while training time increases approximately linearly with D.
  • Classifier comparison: Fig. 8 ranks benchmark MTS classifiers by accuracy, with lower rankings and darker colors indicating better performance.
  • Statistical analysis: The Friedman test found statistically significant performance differences across classifiers, with p-value 1.11e−16.
  • Statistical analysis: Finner post-hoc tests and a Wilcoxon-Holm critical-difference diagram were used to assess pairwise significance.

C. Detailed results on the benchmark datasets

The appendix reports detailed per-dataset results for benchmark MTS classifiers, including accuracy, F1 score, variability, and execution time, alongside ranking and significance visualizations.

  • Detailed benchmark results: Detailed results are provided for the benchmark datasets analyzed in Sections V-A and V-B.
  • Evaluation protocol: Each algorithm was evaluated over 10 independent runs, reporting mean and standard-deviation accuracy, mean and standard-deviation F1 score, and mean execution time.
  • Evaluation caveat: For Arabic Digits, results for 1-NN with DTW are omitted because its simulation exceeded 48 hours.
  • Dataset results: The benchmark includes datasets such as Swedish Leaf, Chlorine Concentration, Distal Phalanx Outline, Electrocardiography, Libras, Character Trajectory, and Wafer.
  • Statistical summaries: Accuracy rankings and post-hoc significance results are summarized through benchmark figures, including the ranking plot, adjusted p-values, and critical-difference diagram.
  • Dataset results: Additional detailed tables report results for Japanese Vowels, Arabic Digits, Australian Sign Language, Network Flow, uWave, Robotic Arm Failure, Performance Measurement System, CMUsubject16, Kick versus Punch, and Walk versus Run.
Loading 1803.07870v3…