PaperlayerView exact revision on arXiv

Source-linked AI summary

Unsupervised Time-Series Representation Learning with Iterative Bilinear Temporal-Spectral Fusion

Ling Yang, Shenda Hong

arXiv:2202.04770v3cs.LGcs.AI

TL;DR

Time-series representation learning lacks sufficient labels, while existing segment-level contrastive methods can lose global context and neglect spectral information. BTSF uses entire-series dropout augmentation and iterative bilinear temporal-spectral fusion, and it significantly outperforms prior methods across classification, forecasting, and anomaly detection.

  • Problem

    Sparse annotations challenge time-series representation learning, while segment-level contrastive methods introduce sampling bias, lose global context, and neglect temporal-spectral relations.

  • Method

    BTSF generates views by applying dropout to entire time series and iteratively fuses temporal and spectral features through bilinear cross-domain affinities.

  • Results

    BTSF significantly outperforms previous methods across classification, forecasting, and anomaly detection and is competitive with supervised methods.

  • Takeaways & Limitations

    The framework provides a unified unsupervised representation-learning approach evaluated across three major time-series downstream tasks.

  • Takeaways & Limitations

    The bilinear feature's quadratic expansion can create memory overhead, motivating a low-rank transformation.

Abstract

from arXiv · show

Unsupervised/self-supervised time series representation learning is a challenging problem because of its complex dynamics and sparse annotations. Existing works mainly adopt the framework of contrastive learning with the time-based augmentation techniques to sample positives and negatives for contrastive training. Nevertheless, they mostly use segment-level augmentation derived from time slicing, which may bring about sampling bias and incorrect optimization with false negatives due to the loss of global context. Besides, they all pay no attention to incorporate the spectral information in feature representation. In this paper, we propose a unified framework, namely Bilinear Temporal-Spectral Fusion (BTSF). Specifically, we firstly utilize the instance-level augmentation with a simple dropout on the entire time series for maximally capturing long-term dependencies. We devise a novel iterative bilinear temporal-spectral fusion to explicitly encode the affinities of abundant time-frequency pairs, and iteratively refines representations in a fusion-and-squeeze manner with Spectrum-to-Time (S2T) and Time-to-Spectrum (T2S) Aggregation modules. We firstly conducts downstream evaluations on three major tasks for time series including classification, forecasting and anomaly detection. Experimental results shows that our BTSF consistently significantly outperforms the state-of-the-art methods.

1. Introduction

Unsupervised time-series representation learning is motivated by sparse labels and complex dynamics, while existing segment-level contrastive methods lose global context and neglect temporal-spectral relations. BTSF addresses these issues through instance-level augmentation and iterative bilinear fusion, with strong downstream results.

  • Sparse labels make unsupervised time-series representations valuable for classification, forecasting, and anomaly detection.
  • Segment-level contrastive methods can introduce sampling bias, false negatives, and lost long-term dependencies by slicing time series.
  • BTSF uses instance-level dropout on entire time series to preserve global temporal information and reduce incorrect contrastive pairs.
  • BTSF proposes iterative bilinear temporal-spectral fusion to model pairwise cross-domain dependencies in a fusion-and-squeeze process.
  • BTSF significantly outperforms previous methods on classification, forecasting, and anomaly detection, while remaining competitive with supervised methods.

2. Related Work

Related work includes reconstruction, clustering, convolutional, and contrastive approaches for unsupervised time-series representation learning. Contrastive methods differ mainly in their segment-level sampling policies but commonly lose global context, risk false negatives, and omit spectral relations.

  • Earlier unsupervised approaches reconstruct inputs, preserve pairwise similarities, combine temporal reconstruction with clustering, or use random convolutional features.
  • Contrastive methods such as CPC, TS-TCC, and TNC construct temporal representations through predictive or neighborhood-based segment sampling.
  • Segment-level contrastive methods are prone to false negatives and fail to capture long-term dependencies because they lose global context.
  • These methods primarily extract temporal features and neglect spectral features and temporal-spectral relations.

3. The Proposed Method

BTSF combines instance-level augmentation with iterative bilinear fusion to preserve global temporal context and model temporal-spectral affinities. It refines temporal and spectral features through cross-domain aggregation while using low-rank factorization to reduce quadratic memory overhead.

  • 3.1. Instance-level Augmentation Technique: Segment-level sampling can lose global semantics and introduce sampling bias, motivating instance-level augmentation for time-series representation learning.The method uses entire time series rather than time slices to preserve long-term dependencies.
  • 3.1. Instance-level Augmentation Technique: BTSF applies independently sampled dropout masks to the entire time series to create positive views while using other variables as negatives in multivariate series.The dropout rate is 0.1 in the experiments, and the construction is intended to preserve the original properties of the series.
  • 3.2. Iterative Bilinear Temporal-Spectral Fusion: BTSF encodes temporal features and FFT-derived spectral features, then models fine-grained affinities between temporal and spectral feature locations.Temporal features use dilated causal convolutions, while spectral features use one-dimensional convolutional blocks.
  • 3.2. Iterative Bilinear Temporal-Spectral Fusion: The bilinear feature summarizes time-frequency interactions and is iteratively used to refine temporal and spectral representations through S2T and T2S aggregation.S2T aggregates spectrum-attentive information into temporal features, while T2S performs the corresponding temporal-to-spectral refinement.
  • 3.2. Iterative Bilinear Temporal-Spectral Fusion: Low-rank factorization reduces bilinear feature storage from O(d^2) to O(ld), where l is much smaller than d.The factorization decomposes the interaction matrix into low-rank mappings before producing bilinear representations.
  • 3.3. Effectiveness of the Proposed BTSF: The contrastive objective separates positive and negative representations, while the bilinear gradient structure couples temporal updates to spectral features and vice versa.The paper uses inner products between ℓ2-normalized feature vectors with temperature parameter τ.

4. Experiments

BTSF is evaluated on classification, forecasting, and anomaly detection using comparable settings and downstream decoders or classifiers. Across these tasks, it achieves strong results, including new state-of-the-art anomaly-detection performance and improved long-series forecasting.

  • Experimental Setup: BTSF is evaluated across classification, forecasting, and anomaly detection, with comparisons to state-of-the-art and supervised methods.The experiments use matched encoder architectures, similar complexity and parameters, and the same representation dimensions where specified.
  • Time-Series Classification: BTSF representations are assessed for classification using a linear classifier, alongside DTW-based nearest neighbors and a supervised comparison model.Evaluation preserves dataset train/test splits and reports prediction accuracy and another metric described in the experimental section.
  • Time-Series Classification: BTSF provides fine-grained temporal-spectral information that supports discriminative downstream classification.The supplied passage identifies temporal-spectral fusion as the source of finer-grained information for discriminativity.
  • Time-Series Forecasting: BTSF achieves the least forecasting error across evaluated prediction settings and performs better as dataset length increases.The reported advantage on longer series is attributed to better use of global context and capture of long-term dependencies.
  • Time-Series Anomaly Detection: BTSF achieves new state-of-the-art F1 scores across all anomaly-detection datasets and surpasses supervised results by a large margin.The authors connect this performance to sensitivity to outliers, long-term dynamics, and fine-grained information from iterative bilinear fusion.

5. Analysis

The analysis examines augmentation, iterative bilinear fusion, representation geometry, and alignment or uniformity. BTSF’s dropout augmentation is reported as accurate and stable, while fusion improves temporal-spectral agreement and representation quality.

  • Comparisons about Time-Series Augmentation Methods: Dropout instance-level augmentation achieves the best average accuracy and variance among the compared augmentation policies.The result is interpreted as evidence that dropout is more accurate and stable for unsupervised time-series representation learning.
  • Impact of Iterative Bilinear Fusion: Adding iterative bilinear fusion raises accuracy and produces 96.60% temporal-spectral overlap, compared with around 30% for existing works.The analysis uses temporal features from S2T and spectral features from T2S to assess cross-domain interaction.
  • Visualization: BTSF representations from the same hidden state cluster better than those from the other approaches in t-SNE visualizations.The analysis treats clustering of same-state representations as evidence that latent information is properly learned and encoded.
  • Visualization: On the UCR archive, BTSF significantly outperforms the other approaches with an average rank of almost 1.3.The result is reported through a critical-difference diagram comparing pairwise statistical differences.
  • Alignment and Uniformity: BTSF achieves the best alignment and an evenly distributed encoding-space feature representation compared with TNC and supervised models.The reported alignment uses positive-pair feature distances, while uniformity concerns preserving information through broad feature distribution.

6. Conclusion

The paper proposes BTSF for unsupervised time-series representation learning by combining instance-level augmentation with iterative bilinear temporal-spectral fusion. Experiments across classification, forecasting, and anomaly detection report superior performance to existing unsupervised methods and the supervised model.

  • Conclusion: BTSF uses entire-series dropout augmentation to preserve global context and generate training views.The conclusion contrasts this instance-level approach with segment-level time slicing.
  • Conclusion: BTSF iteratively refines time-series representations through bilinear temporal-spectral fusion in a fuse-and-squeeze manner.The method is presented as a way to leverage temporal-spectral relations during representation learning.
  • Conclusion: Experiments on classification, forecasting, and anomaly detection report superior BTSF performance, including a large margin over existing unsupervised models and the supervised model.The conclusion summarizes the reported downstream evaluation results without providing task-specific values.

A. More ablation studies

Ablations identify instance-level dropout augmentation and iterative bilinear temporal-spectral fusion as the main contributors to BTSF’s performance. Spectral-temporal integration and iterative refinement provide additional gains over simpler alternatives.

  • 88.3% accuracy is achieved by the TNC baseline using time slicing on HAR.
  • 1.5% improvement over slicing is obtained by layer-wise dropout, while instance-level dropout performs better than segment-level augmentation.
  • Iterative bilinear fusion further improves performance by enhancing and refining temporal-spectral interactions.
  • A dropout rate of 0.1 gives the best performance, whereas higher rates lose raw time-series properties and lower rates risk representation collapse.

B. Datasets descriptions and more experiments

The paper evaluates time-series representations across classification, forecasting, and anomaly detection, using standard task definitions and normalization procedures. Classification accepts uni- or multivariate observations, forecasting predicts future points, and anomaly detection identifies unusual points or subsequences.

  • The experiments cover time-series classification, forecasting, and anomaly detection as downstream tasks.
  • Classification maps univariate or multivariate real-valued observations to a class.
  • Forecasting uses past univariate or multivariate observations to predict future data points.
  • Anomaly detection identifies unusual points or subsequences relative to the series globally or to neighboring points locally.
  • Univariate classification datasets are z-score normalized, multivariate variables are normalized independently, and forecasting metrics use normalized series.

B.1. Classification

Classification experiments use four benchmarks spanning activity, sleep, seizure, and ECG data. BTSF significantly outperforms recent methods and achieves the best reported performance on the Epileptic Seizure Prediction datasets.

  • The classification benchmarks are HAR, Sleep-EDF, Epilepsy Seizure Prediction, and ECG Waveform.
  • HAR contains six activities recorded from 30 subjects using sensors sampled at 50 HZ.
  • Sleep-EDF classifies single-channel overnight EEG signals into five sleep categories.
  • The Epileptic Seizure Prediction dataset contains EEG signals from 500 subjects, with each recording lasting 23.6 seconds.
  • 25 long-term ECG recordings, each 10 hours, comprise the ECG Waveform clinical dataset.
  • BTSF significantly outperforms recent methods and achieves the best performance on univariate and multivariate seizure-prediction classification.

B.2. Forecasting

Forecasting experiments use ETT and Weather data, including long-sequence settings. BTSF achieves the best forecasting results for both short- and long-term ETT prediction by leveraging global context and temporal-spectral relations.

  • Forecasting experiments include four datasets, with two real-world long-sequence datasets and a public benchmark dataset.
  • ETT contains two years of power-system data with oil temperature and six power-load features at hourly and 15-minute granularity.
  • Weather contains four years of hourly climatological data from about 1,600 U.S. places, with wet-bulb temperature and 11 climate features.
  • BTSF achieves the best ETT forecasting results at prediction lengths of 48 and 1440 under both short-term and long-term settings.

B.3. Anomaly detection

The anomaly-detection evaluation uses five publicly available datasets spanning industrial control, server monitoring, and NASA-labeled real-world data. The datasets include normal and attack scenarios where specified, with SMD additionally divided into training and testing subsets.

  • The evaluation covers five publicly available time-series anomaly-detection datasets.The section introduces the datasets as the basis for extensive anomaly-detection experiments.
  • SWaT is a scaled-down industrial water-treatment dataset with 11 days of operation, including 7 normal days and 4 attack-scenario days.
  • WADI extends the SWaT testbed and contains 16 days of operation: 14 normal days and 2 attack-scenario days.
  • SMD contains five weeks of data from 28 server machines, each monitored by 33 metrics, with equal training and testing subsets.
  • SMAP and MSL are publicly available, expert-labeled anomaly-detection datasets from NASA.
Loading 2202.04770v3…