PaperlayerView exact revision on arXiv

Source-linked AI summary

Distributed and parallel time series feature extraction for industrial big data applications

Maximilian Christ, Andreas W. Kempa-Liehr, Michael Feindt

arXiv:1610.07717v3cs.LG

TL;DR

Industrial time-series learning must identify relevant features across multiple series and metadata while coping with distributed data and changing dynamics. FRESH combines comprehensive feature extraction with independently tested, parallel feature filtering. Across benchmark and industrial settings, it preserves classifier performance on most UCR datasets and is applicable to fragmented infrastructures, with redundancy and interaction handling remaining boundaries.

  • Problem

    Feature selection is difficult when industrial classification or regression targets depend simultaneously on several time series and meta-information, especially with distributed data and concept drift.

  • Method

    FRESH extracts comprehensive time-series and metadata features, tests each feature independently for target significance, and applies Benjamini-Yekutieli filtering.

  • Results

    FRESH filtering does not worsen AdaBoost or Random Forest performance on most UCR datasets and is designed for parallel computation on fragmented infrastructure.

  • Takeaways & Limitations

    The algorithm provides a scalable feature-selection approach for industrial, IoT, and Industry 4.0 machine-learning applications with distributed data.

  • Takeaways & Limitations

    FRESH selects associated features redundantly and does not, by design, detect meaningful feature interactions; its performance may also depend on the chosen hypothesis test.

Abstract

from arXiv · show

The all-relevant problem of feature selection is the identification of all strongly and weakly relevant attributes. This problem is especially hard to solve for time series classification and regression in industrial applications such as predictive maintenance or production line optimization, for which each label or regression target is associated with several time series and meta-information simultaneously. Here, we are proposing an efficient, scalable feature extraction algorithm for time series, which filters the available features in an early stage of the machine learning pipeline with respect to their significance for the classification or regression task, while controlling the expected percentage of selected but irrelevant features. The proposed algorithm combines established feature extraction methods with a feature importance filter. It has a low computational complexity, allows to start on a problem with only limited domain knowledge available, can be trivially parallelized, is highly scalable and based on well studied non-parametric hypothesis tests. We benchmark our proposed algorithm on all binary classification problems of the UCR time series classification archive as well as time series from a production line optimization project and simulated stochastic processes with underlying qualitative change of dynamics.

1. Introduction

Industrial machine-learning applications need scalable feature extraction and selection for distributed multivariate time-series data. FRESH combines comprehensive feature mappings with parallel hypothesis-test filtering to retain significant features while controlling irrelevant selections.

  • Industrial sensor-data volumes can prevent centralized processing, while online learning must continually select relevant features under concept drift.
  • FRESH extends prior feature-extraction work with highly parallel filtering for problems combining several time series and meta-information per target.
  • The proposed FRESH algorithm combines comprehensive time-series and meta-information features with independent significance tests and Benjamini-Yekutieli filtering.
  • Evaluation covers binary UCR classification tasks, production-line data, and simulated stochastic processes, with comparisons against established feature-selection and Dynamic Time Warping approaches.
  • The Python implementation, tsfresh, supports fast extraction of many features and integration with scikit-learn, numpy, and pandas.

2. Time series feature extraction

The paper represents multivariate device time series and metadata as lower-dimensional feature vectors for classification or regression. It situates this feature-based approach among shape-based and direct methods, using mappings that capture measurable temporal characteristics.

  • 2.1. Time series: Device data include temporally invariant metadata, irregularly varying information, and regularly sampled sensor measurements forming time series.
  • 2.2. Time series classification: Time-series classification assigns series to discrete categories, whereas regression predicts continuous target values from multivariate time-series input.
  • 2.2. Time series classification: Shape-based methods compare temporal values using distances, while direct approaches learn representations with models such as neural networks, RNNs, RBMs, and CNNs.
  • 2.3. Feature mapping: A feature mapping reduces a time series to a measurable characteristic, such as its maximum, mean, peak count, periodicity, or global trend.
  • 2.3. Feature mapping: The framework maps each of m·n recorded time series through nf feature mappings, producing an m × nφ matrix with nφ = n·nf + ni columns.
  • 2.2. Time series classification: Feature-based classification is established in the literature, with COTE combining 35 classifiers across time, frequency, change, and shape transformations.

3. Feature filtering

Feature filtering defines relevance through statistical dependence, tests each extracted feature with an appropriate non-parametric test, and controls accumulated false selections with Benjamini–Yekutieli. FRESH is scalable and parallelizable, but filter selection can retain redundant correlated features.

  • Relevance of features: A feature is relevant for predicting Y if and only if it is not statistically independent of Y.
  • Feature significance testing: Small p-values indicate features relevant to the target, while the Benjamini-Yekutieli procedure controls the false discovery rate under arbitrary hypothesis dependencies.It orders p-values and rejects those below the threshold determined by the intersection with the rejection line.
  • Hypothesis tests: Specialized non-parametric tests are chosen according to whether the target and feature are binary or non-binary, including Fisher, Kolmogorov-Smirnov, and Kendall rank tests.Kendall’s rank statistic τ measures monotonic association for continuous variables.
  • Feature significance testing: The FRESH algorithm controls the false error rate for all feature and target distributions and dependency structures asymptotically.
  • Hypothesis tests: FRESH extracts features, tests them individually for significance, then applies multiple testing to select relevant features.The procedure uses feature-specific tests rather than one general test for every target-feature combination.
  • The proposed feature extraction algorithm: Filter selection can retain highly correlated features, so FRESH may select or drop correlated statistics such as median and mean together.The proposed FRESH_PCA variant adds a step to avoid groups of highly correlated features.
  • Contribution of this work: FRESH combines established statistical components into a scalable feature-filtering process whose feature calculation and hypothesis tests can be parallelized.

4. Evaluation

The evaluation tests FRESH and variants against other feature-selection pipelines and DTW_NN on benchmark, industrial, and simulated data. FRESH generally preserves or improves accuracy while scaling favorably, although PCA-based filtering and feature removal can lose predictive information.

  • Achieved accuracy: AdaBoost and DTW_NN each achieved the highest mean accuracy on 21 of 32 datasets, with comparable accuracy variance.DTW_NN had higher standard deviation on 9 datasets, while AdaBoost had higher standard deviation on 10.
  • Achieved accuracy: 72% was the highest mean accuracy on iPRODICT, achieved by Boruta_ada and Boruta_rfc; plain AdaBoost reached 71%.DTW_NN reached 57% under a restricted setup using one of the 20 sensor time series and no univariate variables.
  • Achieved accuracy: Feature-based approaches were competitive with DTW_NN, and an unoptimized AdaBoost classifier with generic time-series features performed on par with a state-of-the-art shape-based approach.The evaluation used out-of-the-box classifiers without hyperparameter optimization.
  • Achieved accuracy: FRESH filtering improved or preserved Random Forest accuracy on 22 of 32 datasets and increased it on 14, but reduced accuracy on 13 datasets.Across the compared filters, FRESH most often avoided worsening Random Forest accuracy; feature filtering could nevertheless remove relevant information.
  • Runtime: All feature-extraction methods scaled linearly with sample count and time-series length, unlike DTW_NN.The linear scaling follows from the considered feature mappings having linear runtime in time-series length.
  • Runtime: Only FRESH and FRESH_PCAa scaled linearly with the number of features, making them suitable for filtering large volumes of time-series features.The number of features can increase through additional devices, feature mappings, or time-series types.
  • Selected features: All tested feature-selection methods reduced the number of features used by AdaBoost and Random Forest, while LDA produced the smallest sets.FRESH_PCAa and FRESH_PCAb reduced feature counts further than FRESH because PCA added a second filtering step; FRESH_PCAb reduced them most.

5. Discussion

FRESH is designed as a robust, scalable feature-selection framework for industrial time-series problems, combining comprehensive feature mappings with individually tested feature significance. Its strengths include parallel computation and broad applicability, while its limitations include ignoring feature interactions and potentially retaining correlated features.

  • Motivation: FRESH addresses projects where limited domain knowledge makes early identification of relevant time-series features necessary.It supports subsequent discussions with domain experts and more specialized, domain-specific feature engineering.
  • Operational characteristics: FRESH does not evaluate meaningful feature interactions, although predefined combinations can be incorporated as new features.In the evaluation, Boruta's interaction-aware selection did not outperform FRESH.
  • Scalability: FRESH is highly parallelizable because feature calculations and hypothesis tests can be distributed across computational units and use stateless features.The framework scales linearly with the number of extracted features, time-series length, and considered time series.
  • Feature selection: The method combines established time-series feature mappings with hypothesis-test-based filtering, using the Benjamini-Yekutieli procedure for multiple testing.It can also be applied to features from manifold-structured data such as spectra, images, and videos.
  • Statistical scope: FRESH controls the False Extraction Rate rather than the False Deletion Rate, because the hypotheses required for FDR control are statistically infeasible.Therefore, it is unsuitable when a feature-selection application requires FDR control.
  • Relation to prior approaches: The framework extracts fixed patterns rather than alternating and optimizing feature mappings as some structural or genetic approaches do.Its broader scope includes regression tasks, not only classification.

6. Summary and future work

The paper introduces FRESH as a scalable, parallel feature-extraction and selection method for multivariate time-series classification and regression in distributed industrial settings. Evaluation indicates that filtering generally preserves classifier performance, while the authors identify broader target types and alternative hypothesis tests as future work.

  • Summary: FRESH combines established feature extraction with scalable non-parametric hypothesis-test-based selection using the Benjamini-Yekutieli procedure.It is designed for classification and regression and can incorporate several time-series types plus meta-information per target.
  • Summary: On UCR classification tasks, FRESH filtering did not worsen AdaBoost or Random Forest performance on the majority of datasets.Unfiltered AdaBoost achieved the highest accuracies among the feature-based approaches and beat Dynamic Time Warping nearest-neighbor search.
  • Summary: The method is applicable when data are fragmented across widespread infrastructure and computation cannot be centralized.This follows from parallel feature extraction and filtering, including the demonstrated industrial process-line setting.
  • Future work: The evaluation considered only binary classification variables, leaving multi-class and regression performance for future investigation.The authors also plan to study how alternative hypothesis tests, such as Mann-Whitney U versus Kolmogorov-Smirnov, affect feature significance.
  • Future work: The authors propose examining FRESH in other domains, including image and video processing, because only its feature mappings depend on data structure.

Acknowlegement

The authors acknowledge contributors to the project and implementation and report partial funding from the German Federal Ministry of Education and Research.

  • Acknowledgement: The authors thank named collaborators for discussions and implementation contributions to the tsfresh package.
  • Funding: The research was partly funded by the German Federal Ministry of Education and Research under grant 01IS14004 for project iPRODICT.

Vitae

The authors’ vitae describe careers spanning mathematics, statistics, engineering science, data science, and predictive applications.

  • Author biographies: Maximilian Christ works as a Data Science Consultant at Blue Yonder and pursues a Ph.D. in collaboration with the University of Kaiserslautern.
  • Author biographies: Andreas W. Kempa-Liehr is a Senior Lecturer at the University of Auckland and an Associate Member of Freiburg Materials Research Center.
  • Author biographies: Michael Feindt founded Blue Yonder in 2008 and serves as its Chief Scientific Advisor and Advisory Board member.

Appendix A. Considered feature mappings

FRESH and tsfresh represent each time series through feature mappings that return scalar or vector-valued characteristics. These mappings can include additional parameters, and the evaluated set is a subset of a continuously expanding package.

  • Feature mappings θ_k: R^n_t → R capture characteristics of individual time series.
  • Each mapping takes one time series as input and returns either a real-valued feature or a feature vector whose elements become separate features.
  • Mappings with additional parameters are denoted by θ(S|·), while descriptions omit indices for different devices and sensors.
  • The tsfresh package’s available mappings continuously increase, but the appendix documents those used for the evaluation.

Appendix A.1. Features from summary statistics

The summary-statistics mappings describe central tendency, dispersion, shape, extrema, sample count, and empirical quantiles of a time series. They include both direct statistics and positional or distribution-based summaries.

  • The mappings include sample maximum, minimum, arithmetic mean, variance, standard deviation, skewness, and kurtosis.Kurtosis is described as the fourth central moment divided by squared variance; the cited definition notes a subtraction of 3 for normal-distribution centering.
  • maximum(S) and minimum(S) return the largest and smallest values in S, respectively.
  • length(S) returns the number of samples n_t in the time series.
  • median(S) returns the middle ordered value for odd sample counts or the average of the two middle values for even counts.
  • quantile_of_empiric_distribution_function(S|q) returns the q-quantile, with q% of ordered values lower than or equal to it.

Appendix A.2. Additional characteristics of sample distribution

These mappings characterize distributional energy, stationarity, entropy, variability, symmetry, mass location, and counts relative to mean or median thresholds.

  • absolute_energy(S) interprets S as particle velocity with unit mass and computes its observed energy.
  • The Augmented Dickey-Fuller feature returns the test statistic for the hypothesis that a unit root is present in S.
  • binned_entropy(S|m) divides sample values into m equidistant bins and uses the percentage p_k of samples in each bin.
  • Boolean mappings identify large standard deviation, variance greater than standard deviation, and whether the series appears symmetric.The variance condition is equivalent to the sample variance being larger than 1.
  • mass_quantile(S|q) returns the relative index ν to the left of which q · 100% of the series mass lies; q = 50% gives the mass center.
  • Additional mappings count observations above or below the mean or median.

Appendix A.3. Features derived from observed dynamics

The dynamic-feature mappings capture autoregressive structure, wavelet and Fourier representations, autocorrelation, peaks, spectral density, changes, derivatives, asymmetry, and extrema positions.

  • arima_model_coefficients(S|i, k) fits an unconditional maximum-likelihood AR(k) process and returns coefficient ϕ_i.
  • Continuous-wavelet mappings use the Ricker wavelet; coefficients X(a, b) represent the wavelet of width a at position b.The transform convolves S with the wavelet, and peak detection can inspect coefficients across widths from 1 to l.
  • fast_fourier_transformation_coefficient(S|k) computes the discrete Fourier transform for real input and returns real coefficient parts for natural-number parameters.
  • Other mappings measure first or last extrema positions, lagged or mean autocorrelation, consecutive threshold strikes, changes, derivatives, and time-reversal asymmetry.
  • Peak mappings count peaks by support or compare continuous-wavelet coefficients, while Welch’s method estimates power spectral density from overlapping segments.Welch’s method averages modified periodograms computed for the segments.
Loading 1610.07717v3…