PaperlayerView exact revision on arXiv

Source-linked AI summary

MantisV2: Closing the Zero-Shot Gap in Time Series Classification with Synthetic Data and Test-Time Strategies

Vasilii Feofanov, Songkang Wen, Jianfeng Zhang, Lujia Pan, Ievgen Redko

arXiv:2602.17868v1cs.LGcs.AI

TL;DR

Time-series foundation models still faced data leakage and a substantial gap between frozen and fine-tuned Mantis encoders, limiting confidence in zero-shot classification performance. The paper introduces synthetic-data pre-training, a refined lightweight architecture, and test-time representation strategies, and reports consistent gains over prior time-series foundation models across multiple benchmarks. These results indicate that stronger zero-shot performance can be obtained by improving both the encoder and its test-time use, while further research remains open for multimodal, in-context, and joint classification-forecasting models.

  • Problem

    Prior Mantis pipelines suffered from pre-training data leakage, while frozen Mantis remained substantially behind fine-tuned Mantis in performance.

  • Method

    The paper develops Mantis+ with synthetic-only pre-training, MantisV2 through controlled architectural ablations, and test-time strategies using intermediate representations, token aggregation, self-ensembling, and embedding fusion.

  • Results

    Across UCR, UEA, HAR, and EEG datasets, MantisV2 and Mantis+ consistently outperform prior time-series foundation models in zero-shot evaluation.

  • Takeaways & Limitations

    Zero-shot capabilities of a pre-trained model can be significantly improved by leveraging intermediate-layer representations, refined output-token aggregation, and self-ensembling.

  • Takeaways & Limitations

    The effectiveness of combined output-token aggregation may depend on Transformer depth, and the strategy does not strengthen representations from the last Transformer layer.

Abstract

from arXiv · show

Developing foundation models for time series classification is of high practical relevance, as such models can serve as universal feature extractors for diverse downstream tasks. Although early models such as Mantis have shown the promise of this approach, a substantial performance gap remained between frozen and fine-tuned encoders. In this work, we introduce methods that significantly strengthen zero-shot feature extraction for time series. First, we introduce Mantis+, a variant of Mantis pre-trained entirely on synthetic time series. Second, through controlled ablation studies, we refine the architecture and obtain MantisV2, an improved and more lightweight encoder. Third, we propose an enhanced test-time methodology that leverages intermediate-layer representations and refines output-token aggregation. In addition, we show that performance can be further improved via self-ensembling and cross-model embedding fusion. Extensive experiments on UCR, UEA, Human Activity Recognition (HAR) benchmarks, and EEG datasets show that MantisV2 and Mantis+ consistently outperform prior time series foundation models, achieving state-of-the-art zero-shot performance.

1 Introduction

Time-series foundation models aim to provide general-purpose representations for classification, but prior Mantis models faced data leakage and a substantial frozen-versus-fine-tuned performance gap. MantisV2 and Mantis+ address these issues through synthetic pre-training, architectural refinement, and enhanced test-time processing.

  • Motivation: Time-series foundation models are intended to provide universal feature extractors that reduce labeled-data and model-selection requirements across downstream tasks.Applications span activity recognition, power electronics, observability, healthcare, finance, and neuroscience.
  • Research gap: Prior Mantis pipelines suffered from pre-training data leakage and a substantial performance gap between frozen and fine-tuned encoders.These limitations motivated the development of MantisV2 and Mantis+.
  • Contributions: Mantis+ retrains the original Mantis architecture exclusively on large-scale synthetic time series designed to cover broad temporal patterns.The synthetic-data pipeline is intended to yield more generalizable representations.
  • Contributions: MantisV2 streamlines the Mantis architecture through controlled ablations, producing a more lightweight and more performant encoder.The refinement targets the encoder architecture rather than only its training data.
  • Contributions: The enhanced test-time pipeline combines intermediate-layer representations, output-token aggregation, input-perturbation self-ensembling, and cross-model embedding fusion.These components are reported to improve the robustness and expressiveness of the frozen encoder.
  • Results: Across UCR, UEA, HAR, and EEG datasets, MantisV2 and Mantis+ consistently outperform prior time-series foundation models in zero-shot evaluation.MantisV2 also outperforms TS2Vec and T-Loss, while fusion with a vision backbone can match fine-tuned Mantis.

2 Methodology

The methodology defines time-series foundation models as encoders that produce discriminative embeddings for downstream classification, either with a separate classifier or a fine-tuned head. Mantis processes normalized signals, differentials, and patch statistics into Transformer tokens, and is trained with contrastive learning using augmented views.

  • Problem setup: The foundation model encoder F maps fixed-length time series from R^t into a discriminative hidden space R^q for downstream classification.Embeddings can feed a separate classifier, or a classification head can be fine-tuned with the encoder.
  • Problem setup: For multichannel inputs, Mantis independently encodes each channel and concatenates the resulting embeddings before classification.The alternative of mixing channels with adapters is left for future integration with the proposed variants.
  • Architecture: Mantis fixes the representation to 32 tokens and adapts the Transformer architecture to time series through a dedicated tokenization strategy.The design differs from approaches that fix patch length instead of token count.
  • Token generation: The Token Generator Unit combines instance-normalized signal patches, first-order differential patches, and patch-wise mean and standard-deviation features into 32 projected tokens.Convolution and mean pooling produce the signal and differential patches, while raw-scale statistics preserve measurement information.
  • Transformer: A learnable class token, positional encodings, and six pre-normalized Transformer layers process the generated tokens, with the class-token state serving as the output embedding.The architecture uses eight attention heads per layer; training and fine-tuning append different heads, while inference returns the Transformer embedding directly.
  • Pre-training: Mantis uses contrastive pre-training so augmented views of the same time series receive similar representations while different examples receive dissimilar representations.The model projects embeddings before cosine-similarity computation and samples two augmentations independently from the augmentation set.
  • Pre-training: Random Crop Resize is used for pre-training by cropping a contiguous signal segment and resizing it to the original length, with crop rates sampled from 0% to 20%.The authors select this augmentation because time-series augmentation effectiveness is highly dataset-dependent and aggressive distortions can remove discriminative information.

3 Key Improvements

This section improves zero-shot Mantis representations through synthetic pre-training, architecture ablations, and test-time use of intermediate layers and token aggregation. The resulting choices address data leakage, strengthen representations, and identify settings where intermediate features benefit from increased pre-training scale.

  • Synthetic-data pre-training: Synthetic CauKer data provide an out-of-distribution pre-training setting, avoiding overlap between the original corpus and UCR or UEA evaluation data.CauKer-generated data are OOD by construction, whereas the original 1.89-million-sample corpus partially overlapped with evaluation training sets.
  • Architecture refinement: The Token Generator Unit improves performance by combining raw-signal, first-order-differential, and patch-statistics branches.Each branch contributes positively, and differential features provide a gain beyond increased model capacity; convolution with mean pooling outperforms max pooling and non-overlapping patch embedding.
  • Architecture refinement: A per-head projection dimension of 32 slightly improves average performance while reducing parameters, because the original increase to 128 lacks consistent gains across seeds.The evaluation varies dimensions 32, 64, 128, and 256 and averages results over three random seeds.
  • Architecture refinement: The combination of SwiGLU, RMS Layer Normalization, and RoPE achieves the best UCR performance averaged over three seeds.The improvement is modest but stable across runs and is also confirmed with one million synthetic samples.
  • Layer-wise representations: Intermediate Transformer layers can yield stronger zero-shot representations than the final layer, with Mantis’s third layer achieving the highest accuracy.Layer-wise analysis motivates selecting informative intermediate representations rather than relying only on the final layer.
  • Layer-wise representations: As synthetic pre-training data increase, the best intermediate-layer performance improves consistently while final-layer performance remains unchanged or may degrade.The reported trend indicates that intermediate representations capture scaling benefits, whereas the final layer does not reliably do so.

4 Experimental Results, Part I: Towards Strongest Feature Extractor.

The evaluation compares frozen-encoder zero-shot features across broad time-series benchmarks and model families. MantisV2 leads on UCR and performs strongly across UEA, HAR, and EEG, while runtime and scale vary substantially across baselines.

  • UCR: MantisV2 significantly outperforms other models on average across 128 univariate UCR datasets, with Mantis+ second best.Catch22+ also outperforms NuTime and MOMENT, while tabular models have similar win rates despite lower average performance.
  • UEA-27: Mantis+ and MantisV2 significantly outperform other models by more than 2% on the 27-dataset UEA-27 benchmark.NuTime ranks third on UEA-27 despite performing poorly on UCR.
  • HAR: The gap between MantisV2/Mantis+ and other models is pronounced on human activity recognition tasks.The HAR evaluation includes seven datasets spanning inertial-measurement-unit and motion-capture data, including in-distribution and out-of-distribution HHAR settings.
  • EEG: EEG results show a smaller performance gap between Mantis and other models, while tabular foundation models fail memory or runtime constraints on large datasets.The runtime deadline is 10 hours per method-dataset evaluation.
  • Efficiency: For 100,000 examples, tabular foundation models take more than 10 hours, while vision-based models are slow but feasible.Runtime is measured using forward passes on synthetic univariate series with batch size 256.

5 Experimental Results, Part II: Closing the Zero-Shot Gap.

The paper improves frozen-encoder zero-shot classification at test time through multi-scale and differential-input self-ensembling, cross-model fusion, and classifier choices. These strategies produce measurable gains and help relate MantisV2’s zero-shot performance to fine-tuned and competing systems.

  • Test-time strategies: Test-time optimization combines intermediate-layer representations, output-token aggregation, input perturbations, self-ensembling, and cross-model embedding fusion.The strategies improve frozen representations without additional pre-training or fine-tuning.
  • Self-ensembling: Interpolating the same signal to different lengths yields complementary features because fixed token counts change pooling windows and token overlap.The method independently encodes multiple interpolated versions and concatenates their embeddings.
  • Self-ensembling: Multi-scale interpolation improves UCR performance by 0.66% for Mantis+ and 0.55% for MantisV2.The strategy encodes versions interpolated to lengths 128, 256, 512, and 1024.
  • Self-ensembling: Adding first-order-difference embeddings further improves UCR performance by 0.67% for Mantis+ and 0.29% for MantisV2.The gain occurs even though first-order differential features are already included in the Mantis architecture.
  • Cross-model fusion: Combining MantisV2 embeddings with other models, including Catch22+, improves performance, although gains vary by partner model.The paper also evaluates logistic regression and compares selected fusions with fine-tuned Mantis.
  • Baselines: The extended comparison adds 11 baselines, including self-supervised, supervised deep-learning, and statistical methods.The comparison draws on experimental results from prior studies.

6 Conclusion and Future Work

The paper presents MantisV2 and Mantis+ as synthetic-data-pretrained time-series classification foundation models and reports stronger zero-shot capabilities through test-time strategies. It concludes that further work should address multimodal, in-context, and joint classification–forecasting settings.

  • Conclusion: MantisV2 and Mantis+ are time-series classification foundation models pretrained exclusively on synthetic data.The conclusion presents them as a new generation of Mantis-based models.
  • Conclusion: Intermediate-layer representations, refined output-token aggregation, and self-ensembling significantly improve zero-shot capabilities at test time.The conclusion frames these techniques as improving a pretrained model without changing its pretraining or fine-tuning regime.
  • Future work: Open research directions include multimodal architectures, zero-shot classification through in-context learning, and joint foundation models for classification and forecasting.These directions are presented as future work beyond the paper’s current scope.

A.1.2 HAR Datasets

The HAR evaluation covers seven datasets spanning inertial-measurement-unit and motion-capture recordings, with additional in-distribution and out-of-distribution HHAR settings. The appendix identifies the dataset sources, preprocessing choices, and related HAR subsets in UCR and UEA.

  • Dataset sources: Ego4D supplies inertial-measurement-unit time series preprocessed with the script from Chen et al. (2025).Ego4D is an egocentric multimodal human-activity-recognition dataset requiring a license agreement.
  • Dataset splits: HHAR uses a merged-domain 63.75%-11.25%-25% ID split and an OOD split training on three domains and testing on lgwatch.The OOD training domains are nexus4, s3, and s3mini.
  • Dataset sources: Military Press is represented by MP8 with eight key body-point coordinates and MP50 with all 50 coordinates from 25 body parts.Both versions are used for human exercise performance classification.

A.1.3 EEG Datasets

The EEG evaluation covers datasets from UEA and WOODS, alongside Blink and Epilepsy-EEG, with baseline implementations specified for reproducibility. The experiments also include Catch22-based statistics and several pretrained model baselines.

  • EEG datasets: The EEG experiments include Blink, Epilepsy-EEG, three UEA datasets, and three WOODS benchmark datasets.WOODS covers motor imagery and sleep-stage classification, with separate in-domain and out-of-domain splits.
  • Baseline implementations: Baseline implementations use official or author-provided software and weights, with model-specific preprocessing for NuTime, TabPFN, MOMENT, and vision backbones.NuTime is evaluated without its fine-tuned adapter, TabPFN uses ManyClassClassifier beyond ten classes, and MOMENT processes channels independently.
  • Feature baseline: Catch22+ combines Catch22 statistics with per-patch means and standard deviations as a baseline feature set.Across UCR datasets, this combination improves average accuracy by 3.58%, compared with 0.75% for Catch24.

B Additional Experiments

Additional experiments examine how downstream performance changes across transformer layers, pre-training regimes, and output-token aggregation strategies. They motivate using intermediate representations and combining output tokens in the zero-shot pipeline.

  • Layer representations: Intermediate transformer layers increasingly outperform the final layer on UCR as Mantis receives more updates.The observed pattern suggests that the final layer may overfit the contrastive objective, while intermediate layers generalize better.
  • Pre-training regimes: Layer-by-layer UCR curves over 1000 epochs with 100K samples and over 200 epochs with 2M synthetic samples are used to select final Mantis+ and MantisV2 checkpoints.The 2M-sample curves specifically derive the final checkpoints for both models.
  • Output-token aggregation: Combining the classification token with the mean token improves performance when intermediate representations are used.The comparison also evaluates the same aggregation choices at the last transformer layer, where limited depth may impair classification-token aggregation.

B.3 Architecture Refining

The architecture refinement compares two transformer variants under the same Mantis settings and synthetic-data pre-training. The newer variant produces slightly better UCR performance, while intermediate layers remain necessary for checkpoint selection.

  • Architecture comparison: The second transformer version yields slightly better UCR results than the first when both are pre-trained on 1 million synthetic examples.The comparison changes the transformer architecture while keeping the other Mantis parameters fixed.
  • Layer selection: With 1 million pre-training samples, the final transformer layer is no longer the strongest layer, so all intermediate layers must be considered.The last-epoch UCR accuracy is reported for the architecture comparison.

B.4 Fine-tuning

Fine-tuning experiments test whether truncated encoders and output-token aggregation remain advantageous after supervised adaptation. Keeping the full model is superior, while aggregation provides a smaller positive effect than in zero-shot evaluation.

  • Fine-tuning setup: Fine-tuning appends layer normalization and a linear prediction head to the encoder and optimizes all layers with cross-entropy for 500 epochs.The fixed setup uses AdamW, batch size 128, learning rate 2·10^-4, and weight decay 0.05.
  • Model truncation: Keeping all transformer layers produces better fine-tuning performance than truncating layers after the best zero-shot intermediate layer.Later layers remain important for fine-tuning despite their lower utility during zero-shot feature extraction.
  • Output-token aggregation: Output-token aggregation improves fine-tuning performance for truncated Mantis+ and MantisV2 models.The gain is positive but smaller than the improvement observed in the zero-shot experiment.

C Complete Results

The appendix provides complete zero-shot feature-extraction results across UCR, UEA, HAR, EEG, and broader baseline comparisons, organized by classifier and benchmark.

  • Tables 8–10 report Random Forest zero-shot feature-extraction results for UCR and UEA.
  • Tables 11–16 report Logistic Regression results across UCR, UEA, HAR, and EEG benchmarks.
  • Tables 16–17 present the final comparison on UCR, while Table 18 compares additional baselines across 91 UCR datasets.
Loading 2602.17868v1…