PaperlayerView exact revision on arXiv

Source-linked AI summary

SMart: A Multi-source Multi-phase Time Series Representation Transfer Framework

Fang He, Wang-chien Lee

arXiv:2609.02203v1cs.LGcs.AI

TL;DR

Existing TSRL methods use relatively simple self-recovery tasks and limited source-dataset transfer, while target datasets may be data-sparse. SMart adds multi-phase recurrence-plot recovery and a selector for multiple source datasets, outperforming evaluated baselines across classification and regression.

  • Problem

    TSRL needs richer self-supervised tasks and better source-dataset selection because target datasets can be data-sparse and single-source transfer may be inadequate.

  • Method

    SMart combines three alternative multi-phase recurrence-plot recovery modes with a cross-attention Source Selector that chooses multiple source datasets for encoder pre-training.

  • Results

    SMart outperforms the evaluated baselines across uni-variate and multi-variate classification and regression, reducing MAE by 19.5% and increasing average accuracy by 1.34%.

  • Takeaways & Limitations

    The results support combining dynamic recurrence-plot recovery with learned multi-source selection for time series representation learning and transfer.

  • Takeaways & Limitations

    The paper only explores fundamental recurrence plots with τ=1 and leaves other recurrence plots for future work.

Abstract

from arXiv · show

Time series representation learning (TSRL) has attracted growing research interests in recent years. Two recent explorations in TSRL are: i) exploiting a transformer-based framework to learn time series; ii) instead of using only the targeted dataset, borrowing time series from other datasets to to facilitate representation transfer. While these two explorations are shown effective, the self-supervised time series recovery task in (i) and the single-source dataset used in (ii) are technically simple and thus can be enhanced with new ideas. In this work, we propose a new TSRL framework, namely multi-source multi-phase time series representation transfer (SMart), which has two novel mechanisms to address the aforementioned deficiencies: 1) a multi-phase recurrence plots recovery task, in three alternative modes, for guiding the encoder to embed time series dynamics into the time series representation; and 2) a source dataset selector to select multiple suitable source datasets to supplement the original target dataset for pre-training the TSRL encoder. Experimental results show that SMart outperforms several state-of-the-art models for time series representation learning, classification and regression on both uni-variate and multi-variate time series datasets, reducing mean absolute error up to 19.5% for time series regression, and increasing average accuracy up to 1.34\% for time series classification.

1 INTRODUCTION

SMart extends time series representation learning with multi-phase recurrence-plot recovery and multi-source transfer, addressing limitations in self-recovery tasks and single-source pre-training. It uses a learned selector to choose source datasets and reports improved classification and regression performance.

  • Motivation and contributions: Training data sparsity limits encoder training on target datasets, while DTW-FCN selects only one source dataset using a manually designed similarity metric.The paper argues that multiple suitable source datasets may provide greater benefit than single-source transfer.
  • Method: Three alternative recurrence-plot decoding modes guide the encoder toward multi-granularity time series dynamics.The modes recover recurrence plots in different phase spaces and ways.
  • Method: A cross-attention Source Selector learns dataset relevance from encoder parameters and selects multiple source datasets for the target dataset.The selected sources and target dataset are used to pre-train the time series model before transferring its encoder.
  • Motivation and contributions: SMart combines recurrence-plot recovery with multi-source representation transfer to enrich time series representations and capture dynamics.The framework supplements recovery of the time series itself with recovery of recurrence plots and uses selected source datasets during pre-training.
  • Evaluation: SMart outperforms the evaluated state-of-the-art baselines on both time series classification and regression.The reported contributions include reducing MAE by 19.5% for regression and improving accuracy by 1.34% for classification.

2 RELATED WORK

Related work spans task-specific and general-purpose time series representation learning, including recurrent, convolutional, contrastive, and transformer-based approaches. Existing general-purpose frameworks commonly train on the target dataset alone.

  • Classification and regression: Time series classification and regression research includes both feature-engineered ensembles and neural approaches such as ROCKET.The related-work discussion groups methods by their use of engineered features and model ensembles.
  • Time series representation learning: Time series representation learning methods include task-specific approaches and general-purpose frameworks for unrestricted time series types.The cited applications include human activity, emotion, and speech time series, while other methods target general-type time series.
  • Time series representation learning: General-type TSRL has used multilayer RNNs, LSTMs with attention, dilated CNNs, contrastive learning, and Transformers.These approaches learn representations through sequence reconstruction, attention, contrastive learning, or masked-series recovery.
  • Time Series Transformer: TST masks part of an input time series and decodes its representation to recover the masked segment.This transformer-based self-supervised task is designed to capture useful features in the representation.
  • Representation transfer: The related frameworks described train the encoder using the target dataset only, motivating representation transfer from additional datasets.The supplied passage identifies target-only training as a common characteristic of these state-of-the-art TSRL frameworks.

3 PRELIMINARIES

The preliminaries define TSRL as learning encoders for downstream time series tasks, introduce recurrence plots as dynamic descriptors, and formulate transfer through source-dataset selection. SMart uses these foundations to combine dynamic recovery with transfer learning.

  • 3.1 Problem Formulation: TSRL learns an encoder that maps each time series to a latent representation for downstream applications such as classification and regression.The representation is denoted z_i ∈ R^d in the supplied formulation.
  • 3.2 Recurrence Plots: Recurrence plots record correlations among time-series states at different time points and thereby describe time-series dynamics.They are constructed from phase-space points representing the system status at each time point.
  • 3.2 Recurrence Plots: Phase-space reconstruction uses dimension factor d and time delay τ to form phase points from lagged observations.The phase points are linked in temporal order to form a trajectory whose pairwise distances produce a recurrence plot.
  • 3.2 Recurrence Plots: A recurrence plot is a two-dimensional matrix whose entries are phase-space distances between pairs of phase points.The supplied example uses RP_2,1 with phase points (v_i, v_i+1).
  • 3.3 Source Selection: Time series representation transfer pre-trains an encoder on selected source datasets before transferring it to a target dataset.The source-selection problem is to choose helpful datasets from candidate sources.
  • 3.4 Problem Analysis: SMart treats recurrence-plot recovery and source selection as complementary components of its representation-learning framework.RP recovery guides the encoder toward dynamics at different granularities, while source selection addresses transfer-data choice.

4 FRAMEWORK DESIGN

SMart combines a transformer-based time series model with self-recovery and multi-phase recurrence-plot recovery, then selects multiple source datasets for transfer. Its RP module decodes representations from different encoding layers into recurrence plots, while the Source Selector predicts whether candidate datasets are suitable for pre-training.

  • Time Series Model: SMart extends a time series transformer with self-recovery and RP-recovery decoders to embed both observed values and time series dynamics.The framework uses masked inputs and transformer encoding, with recovery losses applied to the masked portions.
  • RP Recovery: The RP-recovery task decodes representations into RP_1 through RP_K, whose different phase-space dimensions expose time series dynamics at multiple granularities.The method uses mean square error on masked RP values for training.
  • RP Recovery: Because larger-dimensional recurrence plots are harder to recover and smaller-dimensional plots provide hints, the module uses representations from multiple encoding layers.The design is motivated by the progressively richer information associated with larger phase-space dimensions.
  • RP Recovery: The RP-recovery module supports alternative decoding designs, including challenging recovery, recover-in-sequence, and progressive recovery.The supplied passages describe challenging recovery as matching layer k with RP dimension k and recover-in-sequence as progressing from easier to harder plots.
  • Source Selection: The Source Selector treats candidate-source suitability as binary classification and uses first-layer transformer parameters from separately pre-trained models as dataset representations.Selected sources are used with the target dataset to pre-train the time series model before transferring its encoder.

5 PERFORMANCE EVALUATION

The evaluation addresses Source Selector effectiveness, SMart’s performance and efficiency against baselines, and the effects of its components and recovery modes.

  • The experiments evaluate Source Selector design, including dataset representations and attention mechanisms.RQ1 compares first-layer encoder parameters with conventional dataset representatives and tests cross-attention.
  • SMart is evaluated against baseline methods for time series representation learning, classification, and regression.The evaluation also considers efficiency and component-level effects.
  • The study examines pre-training tasks, source-selection strategies, RP-recovery modes, and adaptation to downstream settings.These analyses correspond to RQ3–RQ5.

5.1 Datasets

The evaluation uses 85 bake-off uni-variate datasets and nine labelled UEA multi-variate datasets, adapting the uni-variate data for regression.

  • 85 bake-off uni-variate datasets are collected for time series classification evaluation.For regression, the final value is removed from each series and used as ground truth.
  • Nine labelled multi-variate datasets are collected from the UEA archive.

5.2 Evaluation of Source Selector

The Source Selector is evaluated through grouped dataset splits, attention-design comparisons, and alternative dataset representatives, with cross-attention and early encoder parameters favored by the reported results.

  • 85 datasets are split into 17 groups for training, validation, and testing the Source Selector.Training entries enumerate combined dataset pairs from the 13 training groups.
  • Default outperforms MLP and SelfAtt, supporting cross-attention for source selection.The result is reported both on testing-group pairs and in 17-fold cross-validation.
  • The Source Selector considers encoder parameters and representative time series as alternative dataset representations.RTS-based adaptations include an LSTM selector, while EL-k variants use individual or accumulated encoder-layer parameters.
  • As encoder depth increases, both individual and accumulated EL-k representatives perform worse than earlier-layer representations.Later encoder parameters therefore do not improve dataset representation over first-layer parameters.

5.3 SMart on Uni-variate Time Seires

On uni-variate datasets, SMart combines RP recovery with cross-attention source selection and outperforms strong baselines and ablation variants, while CHR and COR show task-dependent effects.

  • Evaluation against the state of the arts: SMart is evaluated against state-of-the-art classification, regression, and representation-transfer models using selected source datasets for pre-training.The Source Selector chooses sources for each target dataset before TM pre-training.
  • Evaluation against the state of the arts: SMart-RPW reaches 87.84% accuracy, exceeding TST at 86.68% and TS2Vec at 86.66% for classification.The comparison is reported in Figure 6(a).
  • Ablation Study: SMart-RPW consistently outperforms ablation variants, while simply using all source data performs worst among source-selection methods.The results support both RPW recovery and cross-attention source selection.
  • Impacts of CHR and COR: RS outperforms RATE for classification, whereas RATE outperforms RS for regression across source-selection methods.RATE’s final representation includes both CHR and COR, while RS uses CHR only.
  • Impacts of CHR and COR: Adding CHRs raises classification accuracy from 87.65% to 87.75%, while regression MAE drops from 0.315 to 0.314.The reported changes suggest CHRs mainly benefit classification.
  • Impacts of CHR and COR: Adding COR increases classification accuracy and decreases regression MAE, indicating benefits for both tasks.The reported interpretation attributes this to more detailed higher-granularity dynamics in lower RPs.

5.4 Smart for Multi-variate Time Series

SMart is evaluated on multivariate time series using alternative Source Selector training plans and compared with baseline frameworks. Plan 3 performs best, and SMart consistently outperforms the baselines on classification and regression.

  • Source Selector training plans: Plan 3 achieves 79.07% accuracy for classification and 0.26 MAE for regression, outperforming the other Source Selector training plans.The result supports including multivariate time series in the Source Selector’s training group.
  • Source Selector training plans: Including multivariate time series for validation improves source selection, as plan 2 outperforms plan 1 on both tasks.
  • Baseline comparison: SMart consistently outperforms baseline frameworks on multivariate time series classification and regression.The comparison evaluates the quality of representations learned by SMart for both downstream tasks.

6 CONCLUSION

The paper proposes SMart for time series representation learning and transfer, combining recurrence-plot recovery with multi-source dataset selection. Experiments on univariate and multivariate datasets support its effectiveness, while future work targets Source Selector efficiency and broader multivariate training.

  • Contributions: SMart combines an RP-recovery task with a Source Selector that selects proper source datasets for encoder pre-training.
  • Evaluation: Experiments on both univariate and multivariate datasets empirically evaluate SMart’s effectiveness.
  • Future work: Future work will explore more neural network structures to reduce Source Selector time complexity and train it with more multivariate time series datasets.
Loading 2609.02203v1…