PaperlayerView exact revision on arXiv

Source-linked AI summary

Time Series Contrastive Learning with Information-Aware Augmentations

Dongsheng Luo, Wei Cheng, Yingheng Wang, Dongkuan Xu, Jingchao Ni, Wenchao Yu, Xuchao Zhang, Yanchi Liu, Yuncong Chen, Haifeng Chen, Xiang Zhang

arXiv:2303.11911v1cs.LG

TL;DR

Time-series contrastive learning lacks a reliable way to choose meaningful augmentations for diverse data whose temporal structure is difficult to interpret. InfoTS derives information-based criteria for augmentation fidelity and variety and uses a meta-learner to select augmentations. It reports competitive forecasting and classification performance over leading baselines.

  • Problem

    Time-series augmentations are difficult to choose because temporal structures are diverse and hard to interpret, leaving meaningful choices for a task and dataset an open question.

    Source: Introduction · p. 1, Information-Aware Criteria for Good… · p. 3

  • Method

    InfoTS derives information-theoretic criteria favoring high fidelity and variety, then uses a meta-learner to select augmentations for contrastive representation learning.

    Source: Introduction · p. 2, CONCLUSIONS · p. 7

  • Results

    InfoTS reports up to 12.0% reduction in MSE on forecasting and up to 3.7% relative improvement in accuracy on classification over leading baselines.

    Source: Introduction · p. 2

  • Takeaways & Limitations

    The reported experiments support the use of InfoTS representations across downstream forecasting and classification tasks.

    Source: CONCLUSIONS · p. 7

    The fidelity criterion assumes that a time series’ semantic identity is represented by its downstream task label, which may be unavailable during training.

    Source: Information-Aware Criteria for Good… · p. 3

Abstract

from arXiv · show

Various contrastive learning approaches have been proposed in recent years and achieve significant empirical success. While effective and prevalent, contrastive learning has been less explored for time series data. A key component of contrastive learning is to select appropriate augmentations imposing some priors to construct feasible positive samples, such that an encoder can be trained to learn robust and discriminative representations. Unlike image and language domains where ``desired'' augmented samples can be generated with the rule of thumb guided by prefabricated human priors, the ad-hoc manual selection of time series augmentations is hindered by their diverse and human-unrecognizable temporal structures. How to find the desired augmentations of time series data that are meaningful for given contrastive learning tasks and datasets remains an open question. In this work, we address the problem by encouraging both high \textit{fidelity} and \textit{variety} based upon information theory. A theoretical analysis leads to the criteria for selecting feasible data augmentations. On top of that, we propose a new contrastive learning approach with information-aware augmentations, InfoTS, that adaptively selects optimal augmentations for time series representation learning. Experiments on various datasets show highly competitive performance with up to 12.0\% reduction in MSE on forecasting tasks and up to 3.7\% relative improvement in accuracy on classification tasks over the leading baselines.

Introduction

Time-series contrastive learning needs augmentations that preserve meaningful content, but diverse and hard-to-interpret temporal patterns make hand-designed choices difficult. InfoTS defines information-based criteria for suitable augmentations and uses a meta-learner to select them, with competitive forecasting and classification results.

Introduction

  • Time series are complex and often difficult to label, motivating representation learning and contrastive learning as approaches to labeling limitations.The paper describes learned representations as fixed-dimension embeddings intended to retain inherent features with better transferability and generalization capacity than raw series.Source: Introduction · p. 1
  • Time-series augmentation is difficult to hand-design: strong transformations may ruin underlying patterns, while weak ones may be too similar to the input to inform contrastive learning.Time-series datasets also differ across domains, making a universal augmentation or expensive trial-and-error selection potentially suboptimal.Source: Introduction · pp. 1–2
  • Desired augmentations should combine high fidelity, which preserves semantic identity, with high variety, which increases augmentation diversity.The paper derives its selection criteria from information-theoretic analysis and uses one-hot pseudo-labels when downstream labels are unavailable.Source: Introduction · p. 2
  • InfoTS uses a meta-learner to select augmentations from candidates in tandem with contrastive encoder learning.The method is intended to avoid ad-hoc choices and painstaking trial-and-error tuning.Source: Introduction · p. 2
  • Up to 12.0% reduction in MSE on forecasting and up to 3.7% relative improvement in accuracy on classification over leading baselines are reported for InfoTS.The reported contributions describe these results across the paper’s forecasting and classification experiments.Source: Introduction · p. 2

Methodology

InfoTS defines augmentation quality through fidelity to downstream semantics and variety beyond the raw input, then uses an adaptive selector and contrastive encoder to learn representations. Its method combines information-theoretic criteria with global instance-level and local subsequence-level learning.

Notations and Problem Definition

  • The fidelity criterion assumes downstream labels represent time-series semantic identity and seeks high mutual information MI(v; y), approximated by predicting y from augmented v.The prediction projector is optimized with a classification objective.Source: Information-Aware Criteria for Good… · p. 3
  • Preserving one-hot pseudo-labels gives MI(v; y) = MI(x; y) and H(v) ≥ H(x), retaining fidelity while potentially adding information.Pseudo-labels are used when downstream labels are unavailable during unsupervised training.Source: Information-Aware Criteria for Good… · p. 3
  • Variety is modeled by maximizing H(v|x); because unconstrained H(v) may reflect meaningless noise, the method instead minimizes an L1Out upper bound on MI(v; x).The paper notes that other mutual-information upper bounds could also be used in the framework.Source: Information-Aware Criteria for Good… · pp. 3–4
  • A hyperparameter β trades off fidelity and variety; unlike the information bottleneck, these criteria target probabilistic augmentations rather than compressed representations.In the unsupervised setting, the label term uses one-hot pseudo-labels.Source: Information-Aware Criteria for Good… · p. 4

Time Series Meta-Contrastive Learning

  • The encoder combines a fully connected layer with a 10-layer dilated CNN, trained with global instance-level and local subsequence-level contrastive losses.The global loss treats an instance and its augmentation as a positive pair; the local loss forms positive pairs from nearby subsequences and uses non-neighboring samples as negatives.Source: Architecture · p. 4
  • The meta-learner assigns selection probabilities to candidate transformations and approximates discrete Bernoulli choices with binary concrete distributions for gradient-based optimization.The paper states that this makes the meta-network end-to-end differentiable.Source: Meta-learner Network · p. 5

Related Work

Prior time-series contrastive methods use universal or trial-and-error augmentations, while this paper develops an end-to-end differentiable approach to select augmentations for each dataset. It situates that contribution against forecasting architectures and other adaptive-augmentation work.

Contrastive Time Series Representation Learning

  • Existing time-series contrastive methods use universal augmentations or trial-and-error selection, limiting their application to complex real-life datasets.The related work covers segment-discrimination, subsequence-based, neighborhood-based, and hierarchical methods.Source: Contrastive Time Series Representation… · p. 5

Time Series Forecasting

  • Forecasting models span RNN, CNN, Transformer, and GNN architectures, whereas this paper targets general representations compatible with various encoder architectures.The paper distinguishes representation learning for forecasting and other tasks from architectures designed for forecasting itself.Source: Time Series Forecasting · p. 5

Adaptive Data Augmentation

  • Vision methods search for augmentations, but directly applying InfoMin to time series may retain insufficient information; InfoTS differentiably selects augmentations per dataset.The related work describes reinforcement-learning and differentiable policy searches, as well as an Information Bottleneck approach to selecting views.Source: Adaptive Data Augmentation · p. 5

Experiments

InfoTS matches or outperforms leading baselines in forecasting and classification, with reported gains over TS2Vec in both settings. The experiments also examine benchmark protocols and whether the fidelity–variety criteria and adaptive augmentation selection matter.

Experiments

  • The experiments compare forecasting and classification baselines and include case studies of the proposed criteria and meta-learner; full results and sensitivity studies are in the appendix.The main text describes detailed experimental setups and additional experiments as appendix material.Source: Experiments · p. 5

Time Series Forecasting

  • 12.0% average MSE and 9.0% average MAE reductions in univariate forecasting, and 5.5% MSE and 2.3% MAE reductions in multivariate forecasting, versus TS2Vec.The authors report that InfoTS consistently matches or outperforms leading baselines across both settings.Source: Table 1 · p. 6, Time Series Forecasting · p. 6

Time Series Classification

  • 3.7% average classification-accuracy improvement over TS2Vec on 30 UEA datasets for InfoTSs; unsupervised InfoTS ranks second with average rank 2.633.On the 128 UCR datasets, the authors also report that InfoTSs and InfoTS outperform state-of-the-art methods.Source: Table 2 · p. 6, Time Series Classification · p. 7

Time Series Forecasting

  • Forecasting evaluation uses ETTh1, ETTh2, ETTm1, and Electricity in univariate and multivariate settings, with MSE and MAE metrics.Predictions use an L2-regularized linear model, with output dimensions set by forecast horizon and, for multivariate data, feature count.Source: Time Series Forecasting · p. 6

Ablation Studies

  • Figure 3 shows forecasting performance positively related to the proposed criteria in both MAE and MSE settings; subsequence length and jitter scale vary fidelity–variety trade-offs.Shorter subsequences increase variety but can reduce fidelity; increasing jitter standard deviation raises variety and lowers fidelity.Source: Figure 3 · p. 6, Ablation Studies · p. 7

Time Series Classification

  • Classification evaluates UCR's 128 univariate and UEA's 30 multivariate datasets; InfoTS is unsupervised, while InfoTSs uses training labels only for meta-learner selection.Representations are evaluated with an SVM using a radial basis function kernel.Source: Time Series Classification · p. 7

Ablation Studies

  • On Electricity, Random and All are heavily affected by low-quality candidate augmentations, while removing either fidelity or variety confirms both objectives matter.Random selects candidate transformations, All applies them sequentially, and the ablations remove one objective at a time.Source: Table 3 · p. 7, Ablation Studies · p. 7

CONCLUSIONS

InfoTS specifies information-aware criteria for time-series augmentations and uses a meta-learner to select them adaptively. Experiments report state-of-the-art performance in forecasting and classification.

CONCLUSIONS

  • InfoTS proposes that good time-series augmentations preserve both high fidelity and high variety.The criteria are approximated with mutual-information neural estimation and cross-entropy estimation.Source: CONCLUSIONS · p. 7
  • A meta-learner adaptively selects augmentations for contrastive representation learning based on the approximated criteria.Source: CONCLUSIONS · p. 7
  • Experiments report state-of-the-art performance on downstream forecasting and classification tasks.Source: CONCLUSIONS · p. 7

Detailed Proofs

The proofs relate pseudo-label preservation to information retained by augmentations and specify conditions for the criteria. They also describe the binary concrete distribution’s limiting behavior.

Detailed Proofs

  • Preserving one-hot pseudo-labels makes the mutual information between an augmentation and downstream label y equal to that between the raw input and y.Source: Detailed Proofs · p. 10
  • In the unsupervised setting, the augmentation is assumed to be a probabilistic function of x alone, with p(v|x,y) = p(v|x).The proof also describes V(x) as the set of augmented instances of x.Source: Detailed Proofs · p. 10
  • An augmentation that preserves the one-hot pseudo-label is stated to contain at least as much entropy as the raw input.Source: Detailed Proofs · p. 10
  • As temperature τ approaches zero, the binary concrete distribution approaches a Bernoulli distribution.The proof states that the probability of ai = 1 converges to pi.Source: Detailed Proofs · p. 10

Algorithms

InfoTS trains an encoder and meta-learner through alternating contrastive and information-aware objectives. Its local-wise contrastive loss uses neighboring subsequences as positives and more distant ones as negatives.

Training Algorithm

  • For each candidate transformation, InfoTS samples an application decision with a binary concrete distribution and generates augmented instances for the batch.The encoder is updated with global and local contrastive losses, while the meta-learner is updated using fidelity and variety criteria.Source: Training Algorithm · pp. 10–11
  • The classifier is optimized with a cross-entropy classification objective after the encoder and meta-learner updates.Source: Training Algorithm · pp. 10–11

Implementation of Local-Wise Contrastive

  • Local-wise contrastive learning splits an augmented instance into length-L subsequences, treating a neighboring subsequence within window size 1 as positive and those outside it as negative.Source: Implementation of Local-Wise Contrastive · p. 11

Experimental Settings

The experiments use a set of time-series transformations and report implementation and hyperparameter configurations. The settings include augmentation details, hardware, and model-training choices.

Data Augmentations

  • Candidate augmentations include jittering, scaling, cutout, time warping, window slicing, window warping, and subsequence sampling.Examples include adding Gaussian noise, zeroing 10% of sampled timestamps, and cropping half the series before interpolation.Source: Data Augmentations · p. 11

Hardware and Implementations

Hyperparameters

  • The encoder uses 64-dimensional projections and channels, GELU activations, and kernel size 3; Adam uses a 0.001 learning rate, and binary-concrete temperature anneals from 2.0 to 0.1.Trade-off hyperparameters α and β are searched over [0.1, 0.5, 1.0, 5, 10].Source: Hyperparameters · p. 11

More Experimental Results

The experiments examine sensitivity to InfoTS parameters, signal detection, adaptive transformation selection, and performance across tasks. They report benefits from adaptive augmentation, while showing that a useful single transformation can vary by dataset.

Parameter Sensitivity Studies

  • Figure 6 indicates robust performance across tested parameter settings; α or β values of 0.5 or 1 generally achieve good performance.α balances local and global contrastive losses, while β balances variety and fidelity.Source: Figure 6 · p. 13, Parameter Sensitivity Studies · p. 13

Case Study on Signal Detection

  • On CricketX, the 100–200-period augmentation has high fidelity and variety and yields better accuracy than the other tested periods.The study compares subsequence augmentations on periods 0–100, 100–200, and 200–300.Source: Case Study on Signal Detection · p. 13, Figure 7 · p. 13

Updating Process of InfoTS

  • InfoTS assigns increasing weight to subsequence transformation on Electricity, consistent with its strong performance there and its selection as the preferred transformation.The reported comparison also shows InfoTS outperforming the subsequence-only variant.Source: Updating Process of InfoTS · p. 14, Table 5 · p. 14, Figure 8 · p. 14
  • On ETTh1, InfoTS outperforms every single-candidate transformation, while subsequence transformation alone is no more effective than other candidates.The authors attribute the comparison to the meta-learner considering combinations of transformations.Source: Updating Process of InfoTS · p. 14, Table 6 · p. 14
  • For CricketY classification, accuracy is generally positively related to the proposed criteria in both supervised and unsupervised settings.The authors report that this pattern is consistent with the conclusion from forecasting performance.Source: Evaluation of The Criteria with Time… · p. 14, Figure 9 · p. 14

More Ablation Studies

  • InfoTS with TS2Vec as backbone consistently and significantly improves TS2Vec performance on Electricity.The variant uses TS2Vec’s contrastive loss for the encoder and the information-aware criteria to train the meta-learner.Source: More Ablation Studies · p. 15, Table 7 · p. 15

Full Results of Time Series Forecasting and Classification

  • The full-results tables cover multivariate forecasting, univariate classification on 128 UCR datasets, and multivariate classification on 30 UEA datasets.StemGNN results at Ly = 720 are unavailable due to an out-of-memory error; some classification baselines lack scores on datasets with missing observations.Source: Full Results of Time Series… · p. 15, Table 8 · p. 18, Table 9 · p. 19
Loading 2303.11911v1…