PaperlayerView exact revision on arXiv

Source-linked AI summary

TSLANet: Rethinking Transformers for Time Series Representation Learning

Emadeldeen Eldele, Mohamed Ragab, Zhenghua Chen, Min Wu, Xiaoli Li

arXiv:2404.08472v2cs.LGstat.ML

TL;DR

Time-series models must capture long- and short-range dependencies while addressing Transformer sensitivity to noise, computational cost, and small-dataset overfitting. TSLANet replaces self-attention with adaptive spectral and interactive convolution blocks, supported by self-supervised pretraining, and reports superior performance across classification, forecasting, and anomaly detection.

  • Problem

    Transformers capture long-range dependencies but face noise sensitivity, computational inefficiency, and overfitting risks on smaller time-series datasets.

  • Method

    TSLANet uses a lightweight Adaptive Spectral Block with Fourier analysis and adaptive thresholding, an Interactive Convolution Block, and per-dataset self-supervised pretraining.

  • Results

    TSLANet demonstrates superior performance across classification, forecasting, and anomaly detection, including noisy conditions and different data sizes.

  • Takeaways & Limitations

    The experiments support TSLANet as a resilient and adaptable convolutional model for diverse time-series tasks.

Abstract

from arXiv · show

Time series data, characterized by its intrinsic long and short-range dependencies, poses a unique challenge across analytical applications. While Transformer-based models excel at capturing long-range dependencies, they face limitations in noise sensitivity, computational efficiency, and overfitting with smaller datasets. In response, we introduce a novel Time Series Lightweight Adaptive Network (TSLANet), as a universal convolutional model for diverse time series tasks. Specifically, we propose an Adaptive Spectral Block, harnessing Fourier analysis to enhance feature representation and to capture both long-term and short-term interactions while mitigating noise via adaptive thresholding. Additionally, we introduce an Interactive Convolution Block and leverage self-supervised learning to refine the capacity of TSLANet for decoding complex temporal patterns and improve its robustness on different datasets. Our comprehensive experiments demonstrate that TSLANet outperforms state-of-the-art models in various tasks spanning classification, forecasting, and anomaly detection, showcasing its resilience and adaptability across a spectrum of noise levels and data sizes. The code is available at https://github.com/emadeldeen24/TSLANet.

1. Introduction

Transformers capture long-range time-series dependencies but face noise sensitivity, computational inefficiency, and overfitting on small datasets. TSLANet addresses these challenges with lightweight convolutional and adaptive spectral components for diverse tasks.

  • Transformers capture long-range dependencies but can suffer from overfitting, computational inefficiency, and sensitivity to noise, especially on smaller datasets.
  • A 3-layer CNN outperforms Transformer architectures in classification, while its forecasting performance depends on data frequency.The CNN is competitive on Weather’s 10-minute data but struggles on the hourly ETTh1 dataset.
  • TSLANet is a universal lightweight architecture that learns both long- and short-term relationships across time-series tasks.
  • The Adaptive Spectral Block uses global and local Fourier-based filters to cover the frequency spectrum and adaptively attenuate high-frequency noise.
  • The Interactive Convolution Block uses CNNs with different kernel sizes to enrich learning of complex temporal patterns.
  • TSLANet demonstrates superior performance against state-of-the-art methods across classification, forecasting, and other time-series tasks.

2. Related Works

Prior work shows that Transformers model temporal interactions but face concerns about temporal information, computational cost, and small-dataset overfitting, while CNNs capture local patterns but struggle with long-range dependencies. TSLANet combines convolutional local learning with adaptive spectral extraction for broader task coverage.

  • Transformers have been applied to forecasting and anomaly detection, but their temporal modeling, computational demands, and small-dataset generalization have been contested.
  • CNNs are effective for classification and representation learning because they capture local patterns, but their limited long-range modeling challenges forecasting and anomaly detection.
  • Recent CNN-based approaches enhance temporal modeling through frequency analysis, recursive convolution, Fourier-wavelet transforms, or temporal-spectral fusion.
  • TimesNet combines intraperiod and interperiod variations in a 2D representation, but may not fully address non-stationary data without clear periodicity.
  • TSLANet proposes a universal convolutional architecture that combines CNN local feature learning with adaptive spectral extraction for global temporal patterns.

3. Method

TSLANet combines Fourier-domain adaptive filtering with interactive multi-scale convolutions to represent time-series patterns while reducing computational cost. Its pipeline segments inputs into position-aware patches, processes spectral features with ASB, reconstructs them with IFFT, and refines them through ICB.

  • Efficiency: FFT and IFFT provide O(N log N) spectral processing, reducing the computational burden relative to self-attention.The paper contrasts FFT-based computation with the O(N^2) cost of direct DFT computation and describes TSLANet as more efficient than self-attention.
  • Input Patch Embedding: TSLANet segments each multichannel series into patches, projects them to a new dimension, and adds learnable positional embeddings to preserve temporal ordering.Each patch has shape C×p before projection to C×p′.
  • Adaptive Spectral Block: ASB transforms patch representations into the frequency domain with FFT, where each channel is independently represented across spectral characteristics.The transformed sequence length L′ may differ from the original length L depending on the FFT implementation and series properties.
  • Adaptive Spectral Block: A trainable threshold filters frequency components whose power does not exceed θ, adaptively attenuating high-frequency noise while preserving stronger spectral information.The mask retains frequencies with P > θ and removes the others; θ is optimized during training and adapts to dataset characteristics.
  • Adaptive Spectral Block: ASB applies learnable global and local filters to the original and adaptively filtered spectra, integrates their outputs, and uses circular convolution to capture broad periodic structure.The integrated representation is Fintegrated = FG + FL, and the filtering multiplications are equivalent to circular convolution over the sequence.
  • Feature Reconstruction: IFFT converts the integrated spectral features back into time-domain features before they enter ICB for convolutional refinement.The inverse transform reconstructs features aligned with the original time-series structure.
  • Interactive Convolution Block: ICB uses parallel convolutions with different kernel sizes whose outputs interactively modulate one another, combining localized patterns with longer-range dependencies.The activated features are added and passed through a final convolutional layer.

4. Experiments

TSLANet is evaluated across classification, forecasting, and anomaly detection using broad benchmark suites and state-of-the-art baselines. It delivers strong results across tasks, while performance varies among competing model families and dataset characteristics.

  • Experimental Scope: TSLANet is evaluated on classification, forecasting, and anomaly detection tasks across diverse benchmark datasets.The experiments include 116 classification datasets, eight forecasting datasets, and five anomaly-detection datasets.
  • Classification: 83.18% accuracy on UCR and 72.73% on UEA exceed ROCKET’s 81.42% and PatchTST’s 69.38%, respectively.TSLANet also achieves 90.24% overall accuracy on biomedical datasets and 97.46% on human activity recognition datasets.
  • Comparative Findings: Transformer models generally face challenges across datasets, while MLP, TimesNet, and GPT4TS strengths depend on dataset simplicity, frequency content, or size.The reported comparisons associate model-family performance with differing dataset characteristics.
  • Forecasting: TSLANet achieves the second-lowest MSE and MAE on seven of eight forecasting datasets, improving MSE over PatchTST by 3% on ETT(avg) and 3.8% on Weather.Time-LLM performs slightly better overall, but TSLANet remains competitive across datasets.
  • Forecasting: On ETTh1, TSLANet reaches MSE 0.413 versus Time-LLM’s 0.408 while using 6.9e+10 FLOPS versus 7.3e+12.This comparison illustrates similar forecasting performance with substantially lower computational cost.
  • Anomaly Detection: 87.54% overall F1-score makes TSLANet best on most anomaly-detection datasets, including 87.91% on SMD and 97.73% on PSM.GPT4TS ranks second with an overall average F1-score of 86.72%.

5. Model Analysis

Model analyses examine component ablations, noise robustness, scalability, and computational complexity. The results emphasize the Adaptive Spectral Block’s contribution and TSLANet’s efficiency across dataset sizes and model comparisons.

  • Ablation Study: Removing ASB reduces FordA and UWaveGestureLibrary accuracy to 87.3% and 77.5%, and raises ETTh1 and Exchange MSE to 0.421 and 0.380.The ablation results identify ASB as important for feature extraction and noise reduction.
  • Ablation Study: Excluding ASB-L affects noisy datasets more than less noisy datasets, highlighting the local adaptive component’s value for handling noise.ASB-L refers to the local filters in the Adaptive Spectral Block.
  • Ablation Study: Removing ICB reduces classification accuracy and increases forecasting MSE, while removing pretraining slightly diminishes performance across both tasks.The reported degradation from ICB removal is smaller than that from ASB removal.
  • Noise Robustness: With increasing Gaussian noise, Transformer performance deteriorates rapidly, whereas TSLANet remains relatively stable and the Adaptive Filter variant is most resilient.The Adaptive Filter also reduces high-frequency spectral noise spikes while preserving relevant signal.
  • Scalability: On uWaveGestureLibraryAll, TSLANet maintains stable accuracy as layer counts increase, while PatchTST declines markedly, especially at smaller data sizes.TSLANet remains robust as dataset sizes grow, whereas PatchTST tends to decrease at higher layer counts.
  • Complexity Analysis: 77.56% accuracy is achieved with the lowest computational and parameter footprint, including 93% fewer FLOPs and 84% fewer parameters than PatchTST.TSLANet also uses more than 99% fewer FLOPs and parameters than TimesNet while achieving 3% higher accuracy.
  • Complexity Analysis: The reported reductions in computational demand support TSLANet’s lightweight design and more efficient time series analysis.The complexity comparison considers parameters, FLOPs, and accuracy on UEA Heartbeat.

6. Conclusions

TSLANet combines convolution operations with adaptive spectral analysis in a lightweight architecture for time series analysis. Experiments across classification, forecasting, and anomaly detection report strong performance, including under noisy conditions and across data sizes.

  • TSLANet combines convolution operations with adaptive spectral analysis as a lightweight replacement for Transformer-based time series models.
  • Across classification, forecasting, and anomaly detection, TSLANet demonstrates superior performance over traditional Transformer models.
  • TSLANet maintains high accuracy under noisy conditions and across different data sizes.
  • Layer-wise analysis shows advantages on smaller datasets and improved scalability with increasing layers, particularly on larger datasets.

Impact Statement

The paper positions TSLANet as an efficient and scalable approach for time series analysis. Its spectral processing uses adaptive frequency selection and learnable filters before reconstructing time-domain features.

  • TSLANet is proposed as an efficient, scalable, and robust foundation model for time series analysis across applications.
  • Fourier processing decomposes time series into frequencies, with low frequencies representing long-range dependencies and high frequencies representing short-range dependencies.
  • The Adaptive Spectral Block learns frequency filters by adjusting H(f) to minimize a loss measuring discrepancies between output and true data characteristics.
  • The implementation applies FFT, an adaptive high-frequency mask, global and local learnable weights, and inverse FFT in sequence.

D. Experimental Setup

The experiments use task-specific losses, metrics, optimization settings, and dataset splits, with self-supervised pretraining followed by labeled fine-tuning. Training is repeated three times and averaged.

  • Classification uses learning rate 1e-3 and weight decay 1e-4, whereas forecasting and anomaly detection use 1e-4 and 1e-6.
  • Classification runs for 50 pretraining epochs and 100 fine-tuning epochs, while forecasting and anomaly detection run for 10 and 20 epochs.
  • Each experiment uses overlapping windows, is repeated three times, and reports average performance using PyTorch on NVIDIA RTX A6000 GPUs.
  • Classification uses categorical cross-entropy with label smoothing, while forecasting and anomaly detection use Mean Squared Error.
  • Accuracy, MSE and MAE, and F1-score are reported for classification, forecasting, and anomaly detection, respectively.
  • Self-supervised pretraining uses unlabeled training data, followed by fine-tuning on the same data with labels.

E.2. Classification

The classification evaluation spans broad univariate and multivariate archives, biomedical recordings, and human-activity sensor datasets. These datasets vary in domains, modalities, devices, and activity patterns.

  • The evaluation includes UCR, UEA, biomedical, and human activity recognition dataset categories.
  • UCR contains 85 diverse univariate datasets, while UEA contributes 26 preprocessed multivariate datasets.
  • The biomedical evaluation uses Sleep-EDF polysomnography recordings for sleep-stage classification and MIT-BIH ECG recordings for arrhythmia detection and classification.
  • The HAR evaluation uses UCIHAR, WISDM, and HHAR datasets covering smartphone, wearable, and heterogeneous device sensor data.
  • HHAR includes multiple device types and individuals performing activities such as biking, sitting, standing, walking, and stair climbing.

E.3. Forecasting

The forecasting evaluation uses diverse datasets spanning electricity, transformer temperature, exchange rates, traffic, and weather, with varied temporal resolutions and dataset sizes.

  • Forecasting experiments cover electricity consumption, transformer temperature and load, exchange rates, traffic volume, and weather conditions.These datasets represent energy, infrastructure, finance, transportation, and environmental applications.
  • The ETT datasets include hourly and 15-minute sampling intervals, creating forecasting challenges across multiple temporal granularities.The suffixes “h” and “m” denote hourly and 15-minute data, respectively.
  • Table 8 describes each forecasting dataset’s variate count, train-validation-test split sizes, and sampling frequency.

E.4. Anomaly Detection

The anomaly-detection evaluation spans five benchmark datasets covering server monitoring, spacecraft telemetry, earth observation, water-treatment security, and industrial pump sensing.

  • Five anomaly-detection benchmarks represent distinct application areas: servers, spacecraft, environmental monitoring, water treatment, and industrial pumps.The datasets are SMD, MSL, SMAP, SWaT, and PSM.
  • The benchmarks target unusual behaviors linked to failures, security issues, environmental changes, or equipment malfunctions.
  • Table 9 reports each anomaly-detection dataset’s variate count, train-validation-test split sizes, and sampling frequency.

F. Full Results

The full-results section collects classification, forecasting, and anomaly-detection results, while outlining future work on pretraining and noise reduction for TSLANet as a foundation model.

  • Full Results: Tables 10–12 report full classification accuracy results for UCR, UEA, human-activity-recognition, and biomedical-signal datasets.
  • Full Results: Table 13 reports forecasting results across prediction lengths of 96, 192, 336, and 720, using MSE where lower values indicate better performance.
  • Full Results: Table 14 presents the full anomaly-detection results.
  • Future Work: TSLANet is positioned as a potential foundation model, with future work targeting large-scale pretraining and few-shot or zero-shot evaluation.
  • Future Work: The current masking pretraining task may not fully capture complex temporal dependencies and patterns needed for advanced classification and forecasting.
  • Future Work: Future work also proposes noise-reduction methods that handle broader noise distributions and rapid short-term forecasting fluctuations.
Loading 2404.08472v2…