PaperlayerView exact revision on arXiv

Source-linked AI summary

MOMENT: A Family of Open Time-series Foundation Models

Mononito Goswami, Konrad Szafer, Arjun Choudhry, Yifu Cai, Shuo Li, Artur Dubrawski

arXiv:2402.03885v3cs.LGcs.AI

TL;DR

Time-series foundation modeling lacks cohesive public data, handles highly varied datasets, and has limited benchmarks for low-supervision evaluation. MOMENT compiles the Time Series Pile, pre-trains masked-prediction transformers, and evaluates them across practical tasks. The benchmark shows effectiveness with minimal fine-tuning, while the paper also identifies data-quality, fairness, and vertical-shift limitations.

  • Problem

    Time-series modeling lacks a large cohesive public repository, faces heterogeneous data, and has limited benchmarks for diverse tasks under limited supervision.

  • Method

    MOMENT uses high-capacity transformers with masked time-series prediction, pre-trained on the diverse Time Series Pile and evaluated across five tasks in limited-supervision settings.

  • Results

    MOMENT is effective across the considered tasks with minimal fine-tuning, with especially strong performance on anomaly detection and classification problems that typically have small datasets.

  • Takeaways & Limitations

    The paper provides an open-source time-series foundation-model family, public pre-training data, code, model weights, and training logs.

  • Takeaways & Limitations

    MOMENT predictions should be used cautiously in high-stakes settings because reliability depends on training-data quality and diversity, and normalization makes vertical shifts difficult to distinguish.

Abstract

from arXiv · show

We introduce MOMENT, a family of open-source foundation models for general-purpose time series analysis. Pre-training large models on time series data is challenging due to (1) the absence of a large and cohesive public time series repository, and (2) diverse time series characteristics which make multi-dataset training onerous. Additionally, (3) experimental benchmarks to evaluate these models, especially in scenarios with limited resources, time, and supervision, are still in their nascent stages. To address these challenges, we compile a large and diverse collection of public time series, called the Time series Pile, and systematically tackle time series-specific challenges to unlock large-scale multi-dataset pre-training. Finally, we build on recent work to design a benchmark to evaluate time series foundation models on diverse tasks and datasets in limited supervision settings. Experiments on this benchmark demonstrate the effectiveness of our pre-trained models with minimal data and task-specific fine-tuning. Finally, we present several interesting empirical observations about large pre-trained time series models. Pre-trained models (AutonLab/MOMENT-1-large) and Time Series Pile (AutonLab/Timeseries-PILE) are available on Huggingface.

1. Introduction

MOMENT addresses the practical difficulty of general-purpose time-series modeling by introducing open-source foundation models, a diverse pre-training corpus, and a limited-supervision benchmark. The models are designed for multiple tasks and can work with little or no task-specific supervision.

  • Time-series modeling commonly requires substantial domain expertise, time, and task-specific design.
  • MOMENT is an open-source family of high-capacity models for forecasting, classification, anomaly detection, and imputation.The models are intended to serve as building blocks across diverse time-series analysis tasks.
  • MOMENT uses masked time-series prediction to pre-train transformer models on large amounts of data from diverse domains.
  • The Time Series Pile compiles publicly available time-series data from domains including healthcare, engineering, and finance.It was created to address the lack of a large, cohesive public time-series repository.
  • Time-series data vary in temporal resolution, channel count, length, amplitude, and missingness, making large-scale mixed-dataset pre-training challenging.
  • The evaluation covers five modeling tasks against deep-learning and statistical baselines in limited-supervision settings.The tasks are short- and long-horizon forecasting, classification, anomaly detection, and imputation.

2. Related Work

Prior work largely trained time-series models on individual datasets and had limited evaluation in low-supervision settings. MOMENT addresses these gaps through multi-dataset pre-training, masked representation learning, careful data splitting, and broader benchmarking.

  • Transformer self-attention grows quadratically with the length of the input time series.
  • Using disjoint time-series patches as tokens offers an efficient mechanism for learning representations for forecasting.MOMENT builds on a transformer architecture that takes disjoint subsequences as input.
  • Masked representation learning trains models to reconstruct masked input portions and can generalize across downstream tasks.
  • For time series, contrastive learning depends on subjective, data-dependent augmentation, whereas masking reconstructs hidden portions of the signal.
  • MOMENT uses a special mask embedding for time-series patches and pre-trains with masked prediction, supporting forecasting and imputation.
  • To reduce contamination, datasets use creator-defined splits or, when unavailable, random 60% training, 10% validation, and 30% testing splits.
  • The study reports that time-series-pre-trained models outperform LLM-based models on many tasks and datasets.
  • Prior work left the benefits of large-scale multi-dataset pre-training and limited-supervision time-series modeling largely unanswered.

3. Methodology

MOMENT combines the Time Series Pile with a patch-based transformer architecture and masked reconstruction pre-training to support varied time-series inputs. Its methodology also specifies lightweight task adaptation, positional encoding choices, and multiple model scales.

  • 3.1. The Time Series Pile: The Time Series Pile aggregates public time-series repositories spanning diverse domains, lengths, amplitudes, and temporal resolutions.It draws from forecasting, classification, and anomaly-detection collections, including the Informer, Monash, UCR/UEA, and TSB-UAD archives.
  • 3.2. Model Architecture: The model divides time series into disjoint patches, embeds them, replaces randomly selected embeddings with [MASK], and reconstructs patches through a lightweight prediction head.The transformer preserves the D-dimensional patch-embedding shape, and pre-training minimizes masked reconstruction error using Mean Squared Error.
  • 3.2. Model Architecture: MOMENT uses a transformer encoder with relative positional embeddings and adds absolute sinusoidal embeddings to each patch.Subsequent experiments reported that combining absolute and relative positional embeddings improved predictions.
  • 3.2. Model Architecture: MOMENT processes fixed-length univariate inputs by padding shorter series, subsampling longer series, and independently operating on multivariate channels.The fixed input length is T = 512; patching also reduces memory and computational complexity quadratically.
  • 3.2. Model Architecture: A lightweight prediction head supports task-specific fine-tuning while preserving the encoder’s learned parameters and high-level features.This design enables adaptation with a limited number of trainable parameters instead of using a decoder as large as the encoder.
  • 3.3. Pre-training using Masked Time series Modeling: MOMENT is pre-trained in Small, Base, and Large configurations containing approximately 40, 125, and 385 million parameters, respectively.The configurations use 6, 12, and 24 transformer layers, with all weights randomly initialized before pre-training.

4. Experimental Setup and Results

MOMENT is evaluated across five time series tasks in limited-supervision settings, using expanded datasets, multiple metrics, and comparisons against statistical and deep-learning baselines. Results show strong performance across tasks, while analyses examine learned representations, scaling, initialization, and cross-modal transfer.

  • Experimental setup: The benchmark extends prior evaluation across datasets, tasks, and dimensions, including anomaly metrics that account for the sequential nature of time series.Anomaly detection uses adjusted best F1 and VUS-ROC rather than vanilla F1 alone.
  • Experimental setup: The benchmark covers long- and short-horizon forecasting, imputation, classification, and anomaly detection under limited compute and supervision.It evaluates zero-shot and linear-probing configurations alongside task-specific settings.
  • Task results: MOMENT achieves near-state-of-the-art long-horizon forecasting performance on most datasets and horizons, generally behind PatchTST while outperforming LLM-based forecasting models on many settings.N-BEATS also outperforms several recent methods, motivating comparisons beyond transformer-based approaches.
  • Task results: MOMENT shows mixed zero-shot short-horizon forecasting results: statistical methods generally outperform deeper models, but MOMENT achieves lower sMAPE than ARIMA on some datasets.The authors identify this task as having the largest scope for improvement.
  • Task results: Across classification and anomaly detection, MOMENT produces useful representations without dataset-specific fine-tuning and consistently outperforms several competing models on anomaly detection.An SVM using MOMENT representations performs better than all but four specialized time series classification methods, while k-nearest neighbors is marginally better on VUS-ROC but worse on adjusted best F1.
  • Task results: MOMENT with linear probing achieves the lowest reconstruction error on all ETT datasets, and its zero-shot imputation outperforms statistical interpolation methods except linear interpolation.Additional analyses find that the model captures trend, amplitude, frequency, and phase, scaling lowers training loss, random initialization can converge lower, and time-series pre-training supports cross-modal sequence modeling.

5. Conclusion and Future Work

MOMENT combines public data, multi-dataset pre-training, and constrained-resource evaluation to advance open time-series foundation modeling. The authors report effective performance with minimal fine-tuning and identify directions for future work.

  • MOMENT releases an open-source family of time-series foundation models and addresses challenges across data collection, pre-training, and evaluation.The work compiles the Time Series Pile, tackles time-series-specific obstacles to large-scale multi-dataset pre-training, and builds an evaluation benchmark.
  • MOMENT is effective on benchmarked tasks with minimal fine-tuning, with especially strong performance on anomaly detection and classification.The authors attribute this advantage, particularly on small-dataset tasks, to pre-training.
  • The benchmark evaluates multiple practical time-series tasks under constrained compute and supervision.It is designed for scenarios where training or fine-tuning deep neural networks may be infeasible.
  • Smaller statistical and shallower deep-learning methods perform reasonably well across many tasks.
  • Future work includes real-world applications, multimodal time-series and text models, and causal-attention or forecasting-objective pre-training.

Reproducibility statement

The reproducibility statement reports the training hardware and the public availability of selected models, data, and research code. MOMENT-large and the Time Series Pile are available on Hugging Face, while other variants were being prepared for release.

  • All models were trained and evaluated on a cluster with 128 AMD EPYC 7502 CPUs, 503 GB of RAM, and 8 NVIDIA RTX A6000 GPUs.
  • All MOMENT variants were trained on a single A6000 GPU, with any data or model parallelism.
  • MOMENT-large and the Time Series Pile were publicly available on Hugging Face, while MOMENT-base, MOMENT-small, and research code were being open-sourced.

Impact statement

The paper discusses transparency, environmental cost, ethical risks, related modeling choices, and open questions surrounding time-series foundation models. It emphasizes careful deployment, especially where data quality, interpretability, and resource constraints matter.

  • Transparency: MOMENT publicly releases its training code base, data sources, and evaluation pipeline to support transparency.Its expected upstream transparency is reported as one of the highest, while model transparency is lower because harm and trustworthiness evaluations are not well understood for time-series modeling.
  • Environmental Impact: Training multiple models over many days creates significant energy use and a sizeable carbon footprint.The paper estimates emissions using GPU power and grid carbon efficiency, while noting that some other compute power demands are excluded.
  • Environmental Impact: MOMENT-small and MOMENT-base were trained simultaneously on one GPU, whereas MOMENT-large was trained independently on one RTX A6000 GPU.The carbon accounting reports actual consumption for the jointly trained smaller variants and an independent estimate for MOMENT-large.
  • Ethical considerations and potential misuse: MOMENT predictions require care in high-stakes settings because reliability depends on the quality and diversity of training data.The paper warns that unreliable or biased data could produce skewed outputs and unfairly portray or discriminate against groups or individuals.
  • Ethical considerations and potential misuse: MOMENT captures changes in trend, amplitude, and frequency but struggles to distinguish vertically shifted time series after signal normalization.The authors recommend task-specific fine-tuning and in-domain evaluation before high-stakes use, alongside greater interpretability and explainability.
  • Related work and open questions: Prior transformer work addressed quadratic self-attention complexity, while masked prediction offers a reconstruction-oriented pre-training task for time-series applications.The paper also identifies unanswered questions about multi-dataset pre-training and limited-supervision settings.

E. The Time Series Pile

The Time Series Pile aggregates publicly available time-series datasets across many domains and supports MOMENT’s pre-training and evaluation. Its reported scale includes 13 domains, 13 million unique time series, and 1.23 billion timestamps.

  • The Time Series Pile contains 13 unique domains, 20.085 GB of data, 13 million unique time series, and 1.23 billion timestamps.
  • The Time Series Pile covers datasets from 13 distinct domains.
  • The experimental subset includes metadata for M3 and M4 forecasting data and five classification and anomaly-detection datasets.Characteristics of the broader short-horizon forecasting, classification, and anomaly-detection datasets are available in the cited repositories.
  • The anomaly-detection datasets comprise 129,546,401 observations and 1,970 unique time series.
  • The classification datasets comprise 634,084,943 observations and 290,226 unique time series.

F. Experimental Setup and Results

The experiments evaluate MOMENT across multiple time-series tasks under limited and rich supervision, using task-specific configurations and standard benchmarks. Results also examine learned representations and zero-shot reconstruction behavior.

  • Datasets: The benchmark uses diverse forecasting datasets, including short-horizon collections with 281,326,601 observations and 559,102 unique time series.Long-horizon forecasting data include 25,959,994 observations and 1,247 unique time series.
  • Experimental design: MOMENT is evaluated across forecasting, classification, anomaly detection, and imputation in limited- and rich-supervision settings.Limited supervision includes prediction without explicit retraining on target data; rich supervision tests fine-tuning and linear probing.
  • Forecasting: MOMENT supports forecasting through masked-patch prediction, with experiments covering both short- and long-horizon tasks.Long-horizon experiments use a look-back window of L = 512 and evaluate MSE and MAE.
  • Forecasting: Zero-shot short-horizon forecasting is evaluated on M3 and M4 using sMAPE against neural, statistical, and benchmark forecasting methods.The reported evaluation uses the competition train-test splits and includes datasets unseen during pre-training.
  • Classification: For classification, MOMENT can replace its reconstruction head with a linear classifier or use representations in a separate training stage.The classification experiments use 95 UCR datasets containing equal-length univariate series shorter than 512 time steps.
  • Anomaly detection and imputation: Anomaly detection reconstructs the input and flags time steps whose observations differ from predictions beyond a threshold, while imputation masks incomplete patches for reconstruction.The learned mask embedding is approximately standard-normal, and lower-frequency signals are reconstructed better in zero-shot settings.

F.6. Impact of Model Size

Scaling MOMENT generally improves zero-shot performance across forecasting, imputation, and anomaly detection, while broader pre-training data may also help. Representation visualizations further suggest that the model captures several time-series characteristics.

  • Scaling results: Increasing model size generally improves zero-shot forecasting, imputation, and anomaly detection performance.The reported trends are lower MSE and sMAPE and higher VUS-ROC.
  • Scaling results: Increasing the diversity of pre-training data may improve zero-shot performance, based on model checkpoints observed before the first training epoch.The study uses early checkpoints because varying the size of the pre-training dataset is expensive.
  • Learned representations: Embedding visualizations suggest that MOMENT captures trend, scale, frequency, and auto-correlation information in synthetic and real time series.On the five largest UCR datasets, representations form distinct classes without dataset-specific fine-tuning.

H. Results Sources

The reported results draw on task-specific tables, figures, and benchmark summaries covering forecasting, classification, anomaly detection, imputation, scaling, efficiency, and transparency. Across these sources, the paper emphasizes zero-shot and lightly adapted MOMENT performance.

  • Aggregate comparison: The radar plot min-max normalizes methods separately for each of five downstream tasks and reports task-specific aggregate measures.Forecasting and imputation use one minus normalized error, while classification and anomaly detection use average accuracy and VUS-ROC.
  • Classification: Classification tables report accuracy across 91 UCR datasets and 29 UEA datasets, with MOMENT showing promising accuracy without dataset-specific fine-tuning.The accompanying results sources include classification hyperparameters.
  • Imputation: MOMENT achieves state-of-the-art imputation results in both zero-shot and linear-probe fine-tuning settings.The imputation comparison averages results across four masking rates in the patches-missing-at-random setting.
  • Scaling: Scaling tables compare MOMENTsmall, MOMENTbase, and MOMENTlarge for long-horizon forecasting, anomaly detection, and classification.These sources support the model-size analyses described in the paper.
  • Efficiency and transparency: On ETTh1 with prediction horizon 96, MOMENT outperforms the listed forecasting models while using a fraction of the parameters of the most recent LLM-based forecasting method.The paper also reports upstream and model transparency scores, with high upstream transparency but lower model transparency.
Loading 2402.03885v3…