PaperlayerView exact revision on arXiv

Source-linked AI summary

CoST: Contrastive Learning of Disentangled Seasonal-Trend Representations for Time Series Forecasting

Gerald Woo, Chenghao Liu, Doyen Sahoo, Akshat Kumar, Steven Hoi

arXiv:2202.01575v3cs.LG

TL;DR

Time-series forecasting models can overfit unpredictable noise and spurious correlations when learned representations entangle seasonal and trend factors. CoST learns disentangled representations with time- and frequency-domain contrastive losses, then supports downstream forecasting. It achieves state-of-the-art performance, including a 21.3% MSE improvement over the next-best feature-based approach in multivariate benchmarks, and remains robust across backbone encoders and regressors.

  • Problem

    End-to-end forecasting may capture unpredictable noise and spurious correlations, limiting transfer and generalization under non-stationary environments.

  • Method

    CoST learns disentangled seasonal-trend representations using architectural inductive biases, time-domain trend contrastive learning, and frequency-domain seasonal contrastive learning.

  • Results

    21.3% improvement in MSE over the next-best feature-based approach is reported for multivariate benchmarks; CoST also outperforms end-to-end approaches and is robust across backbone encoders and regressors.

  • Takeaways & Limitations

    Separating representation learning from downstream forecasting is presented as a promising alternative to standard end-to-end supervised training for time-series forecasting.

  • Takeaways & Limitations

    The causal formulation assumes observed data are generated from an unpredictable error variable and an error-free latent variable determined by trend and seasonality.

Abstract

from arXiv · show

Deep learning has been actively studied for time series forecasting, and the mainstream paradigm is based on the end-to-end training of neural network architectures, ranging from classical LSTM/RNNs to more recent TCNs and Transformers. Motivated by the recent success of representation learning in computer vision and natural language processing, we argue that a more promising paradigm for time series forecasting, is to first learn disentangled feature representations, followed by a simple regression fine-tuning step -- we justify such a paradigm from a causal perspective. Following this principle, we propose a new time series representation learning framework for time series forecasting named CoST, which applies contrastive learning methods to learn disentangled seasonal-trend representations. CoST comprises both time domain and frequency domain contrastive losses to learn discriminative trend and seasonal representations, respectively. Extensive experiments on real-world datasets show that CoST consistently outperforms the state-of-the-art methods by a considerable margin, achieving a 21.3% improvement in MSE on multivariate benchmarks. It is also robust to various choices of backbone encoders, as well as downstream regressors. Code is available at https://github.com/salesforce/CoST.

1 INTRODUCTION

CoST addresses the limits of end-to-end forecasting by learning disentangled seasonal-trend representations, motivated by causal robustness to distribution shifts. It combines contrastive learning with architectural biases and achieves strong benchmark performance.

  • End-to-end forecasting can overfit unpredictable noise and capture spurious correlations, especially with entangled representations under distribution shifts.
  • Disentangled seasonal-trend representations can improve transfer when one local data-generating module shifts while another remains stable.
  • CoST uses architectural inductive biases to learn separate trend and seasonal representations for long-sequence time-series forecasting.
  • Trend representations use time-domain contrastive learning, while seasonal representations use a frequency-domain contrastive loss.
  • 21.3% improvement in MSE is reported for CoST over existing state-of-the-art approaches in multivariate real-world benchmarks.

2 SEASONAL-TREND REPRESENTATIONS FOR TIME SERIES

The paper models observed series as noisy measurements of latent trend and seasonal factors, then learns representations intended to isolate those factors. This separates representation learning from downstream forecasting and targets robustness to error interventions and distribution shifts.

  • Given a lookback window h, the forecasting task maps h observed timestamps to predictions for the next k time steps.
  • CoST first learns a nonlinear embedding V = f(X), then supplies the final timestamp representation v_h to a downstream forecasting regressor.
  • The causal model treats observed data X as generated from error E and error-free latent X⋆, with X⋆ generated by trend T and seasonality S.
  • Because error E is unpredictable, the optimal prediction is attainable when representations uncover X⋆, which depends only on T and S.
  • Independent seasonal and trend mechanisms remain separately stable when one mechanism changes under distribution shift, supporting transfer in non-stationary environments.
  • Contrastive learning uses augmented data to simulate interventions on the error variable and learn representations for seasonal and trend factors.

3 SEASONAL-TREND CONTRASTIVE LEARNING FRAMEWORK

CoST separates backbone features into trend and seasonal representations, learning them with time- and frequency-domain contrastive objectives. Its architecture combines multi-scale autoregressive trend extraction with a learnable Fourier-based seasonal module.

  • CoST splits intermediate backbone representations into disentangled trend and seasonal features, then concatenates them into final representations.The trend and seasonal dimensions sum to the total representation dimension.
  • 3.1 TREND FEATURE REPRESENTATIONS: The multi-expert trend design addresses lookback-window selection because smaller windows can underfit while larger windows can overfit and over-parameterize.The architecture avoids relying on a single manually selected autoregressive window.
  • 3.1 TREND FEATURE REPRESENTATIONS: The Trend Feature Disentangler uses multiple causal-convolution autoregressive experts with exponentially varying kernel sizes to capture trends across lookback scales.Each expert processes the backbone representation, and their outputs are average-pooled.
  • 3.1 TREND FEATURE REPRESENTATIONS: A time-domain contrastive loss trains discriminative trend representations using augmented positive pairs and negative samples maintained through a momentum encoder and dynamic queue.For each trend sample, a random time step is selected and projected with a one-layer MLP.
  • 3.2 SEASONAL FEATURE REPRESENTATIONS: The Seasonal Feature Disentangler applies a temporal DFT, frequency-specific complex affine transformations, and an inverse DFT to produce seasonal representations.The learnable Fourier layer enables interactions among feature dimensions at each frequency without assuming translation invariance.
  • 3.2 SEASONAL FEATURE REPRESENTATIONS: Frequency-domain contrastive losses use amplitude and phase representations to discriminate seasonal patterns, while augmentations preserve seasonal information by intervening on the error variable.The loss operates on complex-valued pre-inverse-DFT representations, represented through amplitude and phase.

4 EXPERIMENTS

Experiments evaluate CoST across real-world datasets, forecasting settings, backbones, regressors, and component ablations. CoST achieves strong performance and remains robust across alternative architectural and regression choices.

  • Experimental Setup: CoST is evaluated against representation-learning and end-to-end forecasting baselines on five real-world public benchmark datasets.The evaluation covers multivariate and univariate forecasting, using MSE and MAE with train/validation/test splits.
  • Results: 21.3% and 4.71% (MSE) improvements over the next-best feature-based approach are achieved in multivariate and univariate settings, respectively.The authors attribute this performance to learning a composition of trend and seasonal features.
  • Ablation Study: Both trend and seasonal components improve performance over SimCLR and MoCo baselines, while their composition gives the optimal performance.The ablation also reports worse performance when the architecture is trained end-to-end with supervised forecasting loss.
  • Ablation Study: CoST's trend and seasonal components and time- and frequency-domain contrastive losses outperform the competing approach with TCN, LSTM, and Transformer backbones.TCN performs best among the tested backbones, but the proposed approach remains superior in all three settings.
  • Ablation Study: CoST outperforms the competing baseline with ridge, linear, and RBF-kernel ridge regressors.This tests whether the learned representations remain effective across downstream forecasting regressors.

5 RELATED WORK

Related work spans deep end-to-end forecasting, time-series representation learning, and classical decomposition. CoST differs by learning latent representations explicitly composed of trend and seasonal features.

  • Deep Forecasting: Deep forecasting research has used RNNs, Transformers, and other deep models for supervised time-series prediction.Recent Transformer work also addresses the quadratic space complexity of standard Transformer models.
  • Representation Learning: Time-series representation learning has studied contrastive-pair sampling, Transformer architectures, complex contrastive tasks, and temporally hierarchical representations.The cited work had not addressed representations composed of trend and seasonal features.
  • Time-Series Decomposition: Classical decomposition methods separate time series into seasonal and trend components for interpretability, but retain them as signals in the original input space.Recent work has also pursued more robust and efficient decomposition approaches.

6 CONCLUSION

The paper argues that separating representation learning from downstream forecasting is more promising than end-to-end supervised training. CoST learns disentangled seasonal-trend representations and performs well across encoders and regressors.

  • Separating representation learning from downstream forecasting is presented as a more promising paradigm than standard end-to-end supervised training.
  • CoST uses contrastive learning to learn disentangled seasonal-trend representations for time series forecasting.
  • Extensive empirical analysis shows that CoST outperforms previous state-of-the-art approaches by a considerable margin.
  • CoST is robust to various choices of backbone encoders and downstream regressors.

C DATA AUGMENTATIONS

The experiments apply three time-series augmentations in sequence: scaling, shifting, and jittering. Each augmentation is activated with probability 0.5.

  • Three augmentations are composed in order: scaling, shifting, and jittering, each activated with probability 0.5.
  • Scaling: Scaling multiplies every time step by one random scalar sampled from N(0, 0.5).
  • Shifting: Shifting adds one random scalar sampled from N(0, 0.5) to every time step.
  • Jittering: Jittering adds independent Gaussian noise sampled from N(0, 0.5) to each time step.

D SYNTHETIC DATA GENERATION

The synthetic experiments combine generated trend and seasonal patterns into six time series, then compare representation-learning methods under specified training and encoder settings.

  • Synthetic data construction: Two trend patterns are constructed for the synthetic data.
  • Synthetic data construction: Three seasonal patterns use sine waves with periods 20, 50, and 100, alongside specified phases and amplitudes.
  • Synthetic data construction: Each final series is formed by adding a trend pattern and seasonal pattern, producing six series from all pattern pairs.
  • Baseline evaluation: Reported results combine reproduced baselines with results taken directly from prior studies for the ETT, Electricity, and Weather datasets.
  • Encoder settings: All approaches except Triplet and TST use the causal TCN encoder proposed in TS2Vec.
  • Baseline evaluation: TS2Vec, TNC, MoCo, Triplet, and CPC are evaluated using their respective self-supervised representation-learning approaches.
  • Baseline evaluation: TST is a Transformer-based approach using a reconstruction loss, while TCC adds temporal contrastive learning and cross-view prediction.

F RUNTIME ANALYSIS

The runtime analysis reports training and inference times for representation-learning and end-to-end methods. Representation-method runtime separates encoder time from downstream ridge-regressor time.

  • Table 6 reports runtime in seconds for each method during training and inference.
  • For representation-learning methods, runtime is split into encoder time and downstream ridge-regressor time.
  • Experiments are performed on an NVIDIA A100 GPU.
  • Informer runtime may not increase strictly with forecasting horizon because its authors' hyperparameters, including lookback length, vary across horizons.

G UNIVARIATE FORECASTING BENCHMARK

Table 7 presents the univariate forecasting results, with the best results highlighted in bold.

  • Table 7 reports univariate forecasting results.
  • The table highlights the best results in bold.
  • The section evaluates forecasting performance in the univariate setting.

H RESULTS ON FEATURE-BASED FORECASTING BASELINES

The paper compares CoST with feature-based and end-to-end forecasting baselines across univariate, multivariate, and M5 settings. A case study visualizes how its disentanglers separate trend and seasonal information, while TSFresh is limited by high-dimensional univariate feature extraction.

  • Results on feature-based forecasting baselines: Feature-based forecasting results are reported separately for univariate and multivariate settings.
  • Results on feature-based forecasting baselines: TSFresh generally underperforms in the multivariate benchmark because it extracts univariate features, increasing feature dimensionality.The same selected feature set is used across datasets and settings to avoid extensive domain-specific feature engineering.
  • Results on feature-based forecasting baselines: End-to-end forecasting baselines are compared with CoST in both multivariate and univariate experiments.
  • Results on feature-based forecasting baselines: The M5 evaluation uses a multivariate dataset with a forecast horizon of 28, following the competition settings.
  • Case study: disentanglement: TFD representations cluster the two trend patterns more clearly, whereas SFD representations better separate the three seasonal patterns.The visualization fixes seasonality when comparing trends and fixes trend when comparing seasonal patterns.
  • Case study: disentanglement: The synthetic case study combines three seasonality patterns with two trend patterns to evaluate disentanglement.
Loading 2202.01575v3…