PaperlayerView exact revision on arXiv

Source-linked AI summary

SimMTM: A Simple Pre-Training Framework for Masked Time-Series Modeling

Jiaxiang Dong, Haixu Wu, Haoran Zhang, Li Zhang, Jianmin Wang, Mingsheng Long

arXiv:2302.00861v4cs.LG

TL;DR

Random point masking can destroy the temporal variations that carry time-series semantics, making reconstruction too difficult for representation learning. SimMTM reconstructs series from multiple masked neighbors using similarity-based aggregation and manifold constraints, achieving consistent state-of-the-art fine-tuning performance across forecasting and classification in in-domain and cross-domain settings.

  • Problem

    Randomly masking time points can ruin essential temporal variations, making masked reconstruction too difficult to guide time-series representation learning.

  • Method

    SimMTM reconstructs original series from multiple masked neighbors by aggregating point-wise representations using series-wise similarities and a manifold constraint.

  • Results

    SimMTM achieves consistent state-of-the-art fine-tuning performance across forecasting and classification in both in-domain and cross-domain settings.

  • Takeaways & Limitations

    SimMTM provides a masked time-series pre-training framework that covers both low-level forecasting and high-level classification tasks.

  • Takeaways & Limitations

    SimMTM lacks theoretical guarantees for its effectiveness, which the authors leave for future work.

Abstract

from arXiv · show

Time series analysis is widely used in extensive areas. Recently, to reduce labeling expenses and benefit various tasks, self-supervised pre-training has attracted immense interest. One mainstream paradigm is masked modeling, which successfully pre-trains deep models by learning to reconstruct the masked content based on the unmasked part. However, since the semantic information of time series is mainly contained in temporal variations, the standard way of randomly masking a portion of time points will seriously ruin vital temporal variations of time series, making the reconstruction task too difficult to guide representation learning. We thus present SimMTM, a Simple pre-training framework for Masked Time-series Modeling. By relating masked modeling to manifold learning, SimMTM proposes to recover masked time points by the weighted aggregation of multiple neighbors outside the manifold, which eases the reconstruction task by assembling ruined but complementary temporal variations from multiple masked series. SimMTM further learns to uncover the local structure of the manifold, which is helpful for masked modeling. Experimentally, SimMTM achieves state-of-the-art fine-tuning performance compared to the most advanced time series pre-training methods in two canonical time series analysis tasks: forecasting and classification, covering both in- and cross-domain settings.

1 Introduction

Time-series semantics reside mainly in temporal variations, so random point masking can destroy information needed for effective reconstruction. SimMTM addresses this by reconstructing from multiple masked neighbors and learning their manifold structure.

  • Time-series semantic information is mainly contained in temporal variations such as trends, periodicity, and peaks or valleys.
  • Randomly masking time points can seriously ruin these temporal variations, making reconstruction too difficult to guide representation learning.
  • SimMTM reconstructs the original series from multiple randomly masked series rather than directly from unmasked time points.
  • The framework aggregates point-wise representations using similarities learned in a series-wise representation space.
  • SimMTM consistently achieves state-of-the-art fine-tuning performance in forecasting and classification across in-domain and cross-domain settings.

2 Related Work

Self-supervised time-series pre-training builds on contrastive learning and masked modeling. SimMTM extends masked modeling through a manifold perspective that uses multiple masked neighbors for reconstruction.

  • 2.1 Self-supervised Pre-training: Self-supervised pre-training learns generalizable shared knowledge from large-scale data to benefit downstream tasks.
  • 2.2 Contrastive learning: Contrastive learning optimizes representations by bringing positive pairs closer and pushing negative pairs farther apart.
  • 2.2 Contrastive learning: Time-series contrastive methods design pairs or augmentations using temporal properties, including phase shifts, amplitude changes, and patches.
  • 2.3 Masked modeling: Masked modeling reconstructs masked words, image patches, or time points from unmasked content.
  • 2.3 Masked modeling: SimMTM goes beyond classical denoising by projecting masked data back to the original manifold using multiple masked neighbors.

3 SimMTM

SimMTM reconstructs original time series by aggregating multiple masked series using learned series-wise similarities, while manifold constraints guide the representation space.

  • Masking and representation learning: SimMTM generates multiple randomly masked versions of each time series and represents them after encoder and projector processing.Masked values are replaced by zeros, and the masking ratio r and number of masked series M are hyperparameters.
  • Series-wise similarity learning: Series-wise similarities are computed in representation space to avoid the over-smoothing caused by directly averaging masked series.The pairwise similarities are measured using cosine distance.
  • Point-wise reconstruction: Point-wise representations from multiple series are adaptively aggregated according to learned series-wise similarities to reconstruct each original time series.The aggregation includes representations from other series, while the temperature hyperparameter controls softmax normalization.
  • Self-supervised objectives: SimMTM combines reconstruction loss with a manifold constraint that calibrates series-wise representations using assumed neighborhood relationships.The constraint encourages each original series and its masked versions to be close while separating representations from other series.
  • Self-supervised objectives: The overall optimization balances the reconstruction and constraint components, with loss weights adjusted adaptively according to homoscedastic uncertainty.The deep architecture parameters are optimized jointly across the two loss terms.

4 Experiments

SimMTM is evaluated against competitive pre-training baselines across forecasting and classification, using in-domain and cross-domain fine-tuning on twelve real-world datasets. It consistently performs strongly across tasks, transfers across domains, benefits forecasting models and limited-data settings, and depends on both reconstruction and constraint losses.

  • Experimental setup: Experiments cover forecasting and classification under both in-domain and cross-domain settings, using twelve real-world datasets.The comparisons follow established experimental setups and evaluate fine-tuning performance across two task types.
  • Main results: SimMTM significantly outperforms competing baselines across forecasting and classification in both in-domain and cross-domain settings.Figure 3 summarizes performance using forecasting MSE and classification accuracy.
  • Forecasting: 8.3% lower MSE and 4.3% lower MAE than Ti-MAE, and 14.7% lower MSE and 12.0% lower MAE than CoST, are reported on average across forecasting benchmarks.These reductions are measured against the advanced masked-modeling baseline Ti-MAE and contrastive baseline CoST.
  • Forecasting: SimMTM remains overall superior in cross-domain forecasting, although negative migration occurs in some transfers such as Weather → ETTh1 and ETTh2 → ETTm1.The channel-independent encoder supports transfers between datasets with different numbers of variates.
  • Classification: 97.56% versus 81.74% accuracy is reported for SimMTM versus TF-C in the SleepEEG → EMG classification transfer.SimMTM also surpasses other baselines and random initialization significantly in the evaluated cross-domain scenarios.
  • Model analysis: Reconstruction and constraint losses provide 9.7% and 16.0% absolute improvements, respectively, in the SleepEEG → EMG ablation.The constraint loss contributes more and helps uncover a time-series manifold for neighborhood aggregation.
  • Model analysis: SimMTM consistently improves diverse forecasting base models, including PatchTST, and its masking design requires the number of neighbor series to scale with the masked ratio.The model also evaluates fine-tuning under limited-data scenarios.

5 Conclusion

SimMTM is a masked time-series pre-training framework that aggregates multiple masked neighbors using manifold-based similarities. It achieves consistent state-of-the-art fine-tuning performance across forecasting and classification in in-domain and cross-domain settings.

  • SimMTM reconstructs original series from multiple masked neighbors by aggregating point-wise representations according to series-wise similarities.
  • SimMTM constrains series-wise similarities through a neighborhood assumption on the time-series manifold.
  • SimMTM achieves consistent state-of-the-art fine-tuning performance in forecasting and classification tasks across in-domain and cross-domain settings.
  • Experiments are repeated five times using PyTorch on an NVIDIA A100 SXM4 40GB GPU, with MSE and MAE for forecasting and multiple classification metrics.

A.1 Dataset Description

The evaluation uses twelve real-world datasets spanning forecasting and classification, diverse application domains, signal types, and dataset characteristics.

  • The benchmark covers twelve real-world datasets for forecasting and classification across electricity, healthcare, activity recognition, fault detection, and physical-status monitoring.
  • ETT contains four transformer datasets with oil-temperature and power-load series recorded hourly or every 15 minutes.
  • WEATHER contains 21 meteorological indicators collected every 10 minutes in 2020.
  • ELECTRICITY records hourly consumption from 321 Portuguese clients, with 96 measurements per day at 15-minute resolution.
  • SLEEPEEG contains 153 whole-night EEG recordings from 82 healthy subjects, producing 371,055 univariate brainwaves across five sleep-stage categories.
  • EMG contains 163 single-channel recordings sampled at 4K Hz from three volunteers with neuropathy and myopathy.

A.2 Baselines Implementation

The study compares SimMTM with six state-of-the-art baselines using unified and official implementations across diverse pre-training and fine-tuning scenarios. Forecasting and classification use different encoders and task-specific supervised losses.

  • SimMTM is compared with six state-of-the-art baselines using unified-encoder and official-implementation evaluation approaches.
  • The unified setup uses a channel-independent vanilla Transformer for forecasting and 1D-ResNet for classification.
  • Baseline hyperparameters undergo comprehensive searches over encoder depth, model dimension, attention heads, kernel size, and masking settings.
  • The main text reports unified-encoder results for Ti-MAE, TST, CoST, and TS2Vec, but official-code or paper results for LaST and TF-C.
  • The evaluation constructs fifteen in-domain and cross-domain pre-training and fine-tuning experiments across forecasting and classification.
  • SimMTM hyperparameter sensitivity is evaluated on ETTh1 using masked ratio, masked-series count, temperature, masking function, encoder depth, and hidden dimension.
  • A reasonable trade-off between masked ratio and masked-series count is critical, while a small softmax temperature increases differences and masked-sequence diversity.

C Ablations on Aggregation Setting

The ablation compares positive-only and positive-plus-negative neighbor aggregation for reconstructing masked time points. Positive and negative aggregation is consistently stronger than positive-only aggregation.

  • SimMTM reconstructs masked time points through weighted aggregation of multiple neighbors outside the time-series manifold.
  • Positive Samples Aggregation uses only masked series generated from the same sample as reconstruction neighbors.
  • Positive and Negative Samples Aggregation uses masked series from all samples as reconstruction neighbors.
  • PNSA consistently outperforms PSA across all ablation settings, while PSA improves over training from scratch.

D Comparison of Masked Modeling

Figure 6 compares canonical masked modeling with SimMTM’s neighborhood aggregation for reconstructing ETTh1 time series. Direct reconstruction is difficult even at low masking, whereas SimMTM reconstructs the original series precisely.

  • Direct reconstruction is too difficult for TST even at a 12.5% masking ratio and degenerates more seriously at 75%.TST predicts removed time points from the remaining time points.
  • SimMTM precisely reconstructs the original time series through neighborhood reconstruction, supporting its pre-training design.Figure 6 compares TST and SimMTM on ETTh1 cases.

E Full Results

The paper summarizes its experiments through main and analytical results, covering pre-training and fine-tuning for forecasting and classification in in-domain and cross-domain settings.

  • The paper organizes its main-text experiments into main and analytical parts indexed in Tables 13 and 14.
  • Main experiment: The main experiments evaluate pre-training and fine-tuning for forecasting and classification under same- and different-encoder, in-domain and cross-domain settings.
  • Analytical experiment: The analytical experiments examine pre-training and fine-tuning with a unified encoder across forecasting and classification settings.

F Limitations

The paper identifies two limitations: SimMTM lacks theoretical guarantees, and its masking-related hyperparameters still require dataset-specific tuning.

  • SimMTM’s effectiveness is supported empirically, but theoretical guarantees remain absent and are left for future work.
  • The masking ratio and number of masked time series require tuning for different datasets, although M ∝r can ease hyperparameter selection.

G Social Impacts

The paper reports algorithm-focused work conducted under applicable code and dataset licenses, with no identified ethical risk or negative social impact. Its supplementary results span forecasting, classification, ablations, transfer, limited-data fine-tuning, and multiple encoder settings.

  • Overall results: The paper states that SimMTM achieves state-of-the-art performance in two mainstream time series analysis tasks.
  • Social impacts: The paper focuses only on algorithm design and states that its code and datasets follow the corresponding licenses.
  • Social impacts: The authors report no potential ethical risk or negative social impact.
  • Forecasting results: Supplementary forecasting results cover in-domain and cross-domain settings, unified encoders, ablations, transfer, negative transfer, and limited-data fine-tuning.
  • Classification results: Supplementary classification results cover in-domain and cross-domain settings, unified 1-D ResNet encoders, and ablations across four datasets.
Loading 2302.00861v4…