PaperlayerView exact revision on arXiv

Source-linked AI summary

Unified Training of Universal Time Series Forecasting Transformers

Gerald Woo, Chenghao Liu, Akshat Kumar, Caiming Xiong, Silvio Savarese, Doyen Sahoo

arXiv:2402.02592v2cs.LGcs.AI

TL;DR

Time-series forecasting remains largely organized around one model per dataset, limiting adaptation across diverse datasets and forecasting scenarios. This paper introduces MOIRAI, a masked encoder-based universal Transformer trained on LOTSA, and reports competitive or superior zero-shot performance versus full-shot models.

  • Problem

    Time-series forecasting lacks a single model that can generalize across diverse datasets and forecasting scenarios.

  • Method

    MOIRAI is a masked encoder-based Transformer trained on LOTSA, a 27B-observation archive across nine domains, with components supporting cross-frequency and arbitrary-variate forecasting.

  • Results

    MOIRAI achieves competitive or superior performance as a zero-shot forecaster compared with state-of-the-art full-shot baselines in in-distribution and out-of-distribution evaluations.

  • Takeaways & Limitations

    The results support MOIRAI as a universal forecaster capable of probabilistic and long-sequence forecasting across in-distribution and out-of-distribution settings.

  • Takeaways & Limitations

    MOIRAI’s cross-frequency multi-patch-size mapping is heuristic, and its architecture has limited support for high-dimensional time series.

Abstract

from arXiv · show

Deep learning for time series forecasting has traditionally operated within a one-model-per-dataset framework, limiting its potential to leverage the game-changing impact of large pre-trained models. The concept of universal forecasting, emerging from pre-training on a vast collection of time series datasets, envisions a single Large Time Series Model capable of addressing diverse downstream forecasting tasks. However, constructing such a model poses unique challenges specific to time series data: i) cross-frequency learning, ii) accommodating an arbitrary number of variates for multivariate time series, and iii) addressing the varying distributional properties inherent in large-scale data. To address these challenges, we present novel enhancements to the conventional time series Transformer architecture, resulting in our proposed Masked Encoder-based Universal Time Series Forecasting Transformer (Moirai). Trained on our newly introduced Large-scale Open Time Series Archive (LOTSA) featuring over 27B observations across nine domains, Moirai achieves competitive or superior performance as a zero-shot forecaster when compared to full-shot models. Code, data, and model weights can be found at https://github.com/SalesforceAIResearch/uni2ts.

1. Introduction

The paper motivates universal forecasting as a shift from dataset-specific models toward a single pre-trained forecaster, but identifies heterogeneity in frequencies, variates, distributions, and available training data as key obstacles. It addresses these challenges with a modified Transformer, LOTSA pre-training, and MOIRAI, which achieves competitive or superior zero-shot performance against full-shot models.

  • Motivation: Universal forecasting replaces fixed-context, fixed-horizon, single-dataset training with a single large pre-trained model intended to generalize across diverse forecasting tasks.The existing paradigm is described as increasingly antiquated because it lacks capacity to generalize or adapt across scenarios and datasets.
  • Challenges: Time series heterogeneity creates three core challenges: cross-frequency learning, arbitrary-variate multivariate forecasting, and flexible probabilistic modeling across differing distributions.Frequency affects temporal patterns; universal models must capture multivariate interactions and exogenous covariates; fixed symmetric predictive distributions may not suit positive series.
  • Data and training: 27B observations across nine domains power LOTSA, a large-scale open archive introduced for pre-training large time series models.The paper also introduces UNI2TS, model weights, and LOTSA as open-source resources.
  • Method: The proposed Transformer modifies a masked encoder with multiple input and output projection layers to handle time series with varying frequencies and arbitrary heterogeneity.The architecture is designed to alleviate the identified issues, while its components are intended to extend beyond masked encoders to other Transformer variants.
  • Results: MOIRAI achieves competitive or superior zero-shot forecasting performance compared with full-shot models after training on LOTSA data.The model is trained by optimizing the negative log-likelihood of a mixture distribution and randomly sampling context and prediction lengths for flexible downstream use.

2. Related Work

Recent work has introduced pre-trained models for zero-shot time series forecasting, while earlier pre-training methods primarily focused on fine-tuning within the same dataset. These approaches differ in model accessibility, data sources, architectures, and multivariate or cross-frequency handling.

  • Pre-training for Zero-shot Forecasting: TimeGPT-1 introduced a closed-source model supporting zero-shot forecasting and API-based fine-tuning, currently limited to beta users.
  • Pre-training for Zero-shot Forecasting: ForecastPFN pre-trains on synthetic time series, while related models use large-scale Google Trends, Wiki pageviews, and open-data collections.
  • Pre-training for Zero-shot Forecasting: Tiny Time Mixers uses a lightweight mixer architecture, downsampling for augmentation, and an exogenous mixer to support multivariate downstream tasks through fine-tuning.
  • Denoising autoencoders and contrastive learning improved forecasting as pretext tasks, but were largely evaluated through same-dataset pre-training followed by fine-tuning rather than generalization.

3. Method

Moirai is an encoder-only, patch-based Transformer that forecasts distribution parameters from historical targets and covariates by maximizing mixture-distribution likelihood. Its design addresses universal forecasting through multi-patch processing, arbitrary-variate handling, flexible output distributions, and task-aware pre-training on LOTSA.

  • Problem Formulation: Moirai predicts forecast-distribution parameters from target and covariate histories using a learned model optimized by log-likelihood.The formulation uses context length l and prediction length h to define the forecasting task.
  • Architecture: Moirai uses non-overlapping patches, multi-patch-size input and output projections, forecast-horizon mask embeddings, and instance normalization within a masked encoder architecture.The output projection maps decoded tokens to mixture-distribution parameters.
  • Cross-Frequency Learning: Multi-patch-size processing accommodates diverse frequencies while larger patches reduce quadratic attention costs for high-frequency data.The strategy is designed to preserve a long context while lowering attention computation.
  • Any-Variate Modeling: Flattening all variates into one sequence enables arbitrary multivariate dimensionality, while variate encodings and binary attention bias distinguish variates and preserve permutation equivariance or invariance.The binary attention bias also supports extension to an arbitrary number of variates.
  • Distributional Modeling: The output distribution is a flexible mixture of parametric components, including Student’s t, negative binomial, and log-normal distributions.These components target robust general forecasting, positive count data, and commonly right-skewed data, respectively.
  • Pre-Training: Pre-training samples sub-datasets and conditional time series from LOTSA, then varies context and prediction lengths through a task distribution to support diverse downstream tasks.The data distribution is decomposed into sub-dataset and conditional time-series distributions, while the task distribution samples forecasting windows.

4. Experiments

Experiments show that MOIRAI performs strongly as a unified model across in-distribution and zero-shot out-of-distribution forecasting tasks. Ablations further indicate that its architecture, diverse pre-training data, flexible distributions, and packed training contribute to performance and scalability.

  • In-distribution Evaluation: MOIRAI outperforms all Monash benchmark baselines across model sizes while using one model across datasets rather than dataset-specific models.This demonstrates strong in-distribution and cross-domain performance from unified training.
  • Probabilistic Forecasting: MOIRAIBase and MOIRAILarge achieve best or second-best probabilistic zero-shot results on every evaluated dataset except Walmart and Istanbul Traffic.Performance remains close to the best on those exceptions despite comparison with tuned full-shot baselines.
  • Long Sequence Forecasting: MOIRAIBase consistently delivers strong long-sequence forecasting, while MOIRAILarge is less consistent but remains competitive and shows a tenuous size–performance relationship.Results are averaged across prediction lengths and compared with full-shot baselines.
  • Architecture: Removing multi-patch sizing, Any-variate Attention, or mixture distributions deteriorates normalized MAE or probabilistic forecasting quality, supporting these architectural components.Student’s t-distributions can produce inappropriate symmetric intervals for peaks.
  • Training Methodology: Diverse pre-training data is critical for cross-domain performance, while packed training significantly improves performance by increasing effective batch size and observed training data.Packing also addresses longer training times and sequence-length disparities in the flattened setting.
  • Context Length: MOIRAI can handle arbitrary context lengths, including thousands of time steps, in zero-shot forecasting.The capability is evaluated by relating performance to increasing context lengths across three datasets.

5. Conclusion

The paper introduces MOIRAI, a masked encoder-based universal time series forecasting Transformer, alongside LOTSA, a large open-data collection for pre-training forecasting models. MOIRAI is evaluated across in-distribution and out-of-distribution settings and supports probabilistic and long-sequence forecasting, while limitations remain in tuning, cross-frequency design, and high-dimensional inputs.

  • Conclusion: MOIRAI is a masked encoder-based universal time series forecasting Transformer designed to alleviate challenges in universal forecasting.The work also introduces LOTSA, described as the largest collection of open data for pre-training time series forecasting models.
  • Conclusion: MOIRAI is evaluated in both in-distribution and out-of-distribution settings and supports probabilistic and long-sequence forecasting.The paper reports that MOIRAI achieves competitive performance as a zero-shot forecaster, although the supplied passage is truncated after this claim.
  • Limitations & Future Work: The authors characterize MOIRAI as a first step in the universal forecasting paradigm despite its in- and out-of-distribution performance.This frames the reported results as motivation for continued work rather than a final solution.
  • Limitations & Future Work: The authors identify limited hyperparameter tuning, heuristic multi-patch-size cross-frequency mapping, and limited support for high-dimensional time series as limitations.They attribute the tuning constraint to limited resources and call for more flexible cross-frequency methods and efficient extensions of Transformer input length.

A. Large-scale Open Time Series Archive

LOTSA is a curated collection of time series datasets assembled for pre-training large time series models, spanning diverse domains, frequencies, variates, covariates, and dataset sizes. It combines established libraries and benchmarks with additional preprocessing and dataset-specific splitting procedures, totaling 231B observations when variates are counted.

  • Archive scope: LOTSA curates time series datasets for pre-training large time series models and records domain, frequency, series, variate, covariate, and observation properties.The archive documents preprocessing and data splitting for its constituent datasets.
  • Archive scale: 231B total observations are counted when each time-series observation is multiplied by its number of variates.The reported total is 231,082,956,489 observations.
  • Constituent sources: The archive draws from BuildingsBench, ClimateLearn, CloudOps TSF, GluonTS, LargeST, LibCity, Monash, ProEnFo, and SubseasonalClimateUSA.These sources cover building energy, climate, cloud operations, general forecasting, traffic, urban spatio-temporal, load, and subseasonal climate data.
  • Preprocessing and splits: Dataset handling uses source-specific splits, including pre-train-only CloudOps data, train-region Monash data, and dropped spatial structure for LibCity.The Monash final forecast horizon is held out for in-distribution evaluation, while CloudOps test sets are held out.
  • Preprocessing and splits: Miscellaneous raw datasets undergo filtering for overly short series or excessive missing values, with all variates treated as targets unless dataset creators specify otherwise.These datasets require more extensive preprocessing because they are not provided through a library or collection.

B. MOIRAI Architecture Details … C. Probabilistic Forecasting

Moirai uses frequency-specific patch projections with shared layers across overlapping frequencies and predicts mixture-distribution parameters for probabilistic forecasts. Its distributional design combines constrained Student’s t, log-normal, negative-binomial, and low-variance normal components, differing in flexibility from related models.

  • B.1. Multi Patch Size Projection Layers: Moirai heuristically maps lower-frequency data to smaller patches and higher-frequency data to larger patches through multi-patch-size projection layers.Each input projection maps patch size to hidden state, while each output projection maps hidden state to distribution parameters.
  • B.1. Multi Patch Size Projection Layers: Monthly series use patch sizes 8, 16, 32, while weekly and daily series use 16, 32.The frequency mapping assigns fewer, larger patch sizes as frequency increases from monthly to weekly and daily data.
  • B.1. Multi Patch Size Projection Layers: Moirai learns one input and one output Linear layer per patch size, sharing layers across overlapping frequencies, for five of each layer type.The sharing scheme reduces the number of distinct projection layers across frequency-specific patch configurations.
  • B.2. Mixture Distribution: For probabilistic forecasting, Moirai predicts mixture-distribution parameters and applies softmax to mixture weights, constraining them to the probability simplex.The mixture components include Student’s t, log-normal, negative binomial, and low-variance normal distributions.
  • B.2. Mixture Distribution: Student’s t components predict degrees of freedom, location, and scale with softplus positivity constraints, while lower-bounding degrees of freedom to 2.The lower bound ensures the variance is defined.
  • B.2. Mixture Distribution: Log-normal components predict location and scale with softplus positivity constraints, whereas continuous negative-binomial components use softplus for r and sigmoid for p.The negative-binomial implementation follows a continuous extension and constrains p to a probability.
  • B.3. Discussion on “Flexible Distribution”: Compared with related probabilistic models, Lag-llama uses a symmetric Student-T distribution, TimeGPT uses conformal intervals that may violate domain constraints, and LLMTime uses a categorical distribution.The paper categorizes Lag-llama and TimeGPT as inflexible, while describing LLMTime’s categorical approach as able to approximate many continuous distributions.

C.1. Evaluation Metrics

The section defines CRPS as a probabilistic forecasting metric and uses a normalized discrete quantile-loss approximation in practice. It also introduces MSIS for evaluating uncertainty around point forecasts using prediction intervals.

  • Continuous Ranked Probability Score: CRPS evaluates probabilistic forecasts from a predicted distribution F and ground truth y.It is defined using the continuous ranked probability score formulation.
  • Continuous Ranked Probability Score: The quantile-loss term Λα is the pinball loss evaluated at quantile level α.This links the CRPS formulation to quantile-based evaluation.
  • Continuous Ranked Probability Score: Because CRPS can be intractable or expensive and normalization is desired, the method uses mean weighted sum quantile loss as a discrete approximation.The approximation averages quantile losses across K quantiles.
  • Continuous Ranked Probability Score: In practice, K = 9 quantiles span α = 0.1, 0.2, ..., 0.9, using predicted α-quantiles at each forecast step.The predicted quantile at time step t is denoted ˆqt(α).
  • Mean Scaled Interval Score: MSIS evaluates uncertainty around point forecasts from upper and lower prediction bounds.For a 95% prediction interval, the significance level is a = 0.05 over forecast horizon h, with m as the seasonal factor.

C.2. Evaluation Setup · C.3. Baselines

The evaluation uses non-overlapping rolling windows across four domains and frequencies, with train, validation, and test sets defined relative to the forecast horizon. Deep learning baselines are tuned through random search over prescribed hyperparameters, learning rates, and context lengths across 15 training runs, with results reported on five independent train.

  • C.2. Evaluation Setup: The probabilistic forecasting evaluation uses a dataset summary covering the out-of-distribution setting.Table 18 summarizes the datasets used in this evaluation.
  • C.2. Evaluation Setup: Evaluation spans four domains ranging from minute-level to weekly frequencies.The datasets are summarized in Table 18 for the out-of-distribution probabilistic forecasting setting.
  • C.2. Evaluation Setup: The evaluation uses non-overlapping rolling windows with stride equal to the prediction length.The test set comprises the last h ∗r time steps, where h is the forecast horizon and r is the number of rolling windows.
  • C.2. Evaluation Setup: The validation set is the forecast horizon immediately preceding the test set.The train set contains everything before the validation set, while the test set contains the final rolling-evaluation windows.
  • C.3. Baselines: The four deep learning baselines are DeepAR, PatchTST, TiDE, and TFT.Their hyperparameter search values are presented in Table 19.
  • C.3. Baselines: Baseline tuning includes learning rates from 1e-6 to 1e-3 on a logarithmic scale.The search also tunes context length as l = m ∗h, with m ranging from 2 to 20 and h equal to prediction length.
  • C.3. Baselines: Random search explores the tuning values over 15 training runs.The procedure reports results on 5 independent train.

D. Full Experimental Results … D.3. Out-of-distribution Forecasting: Long Sequence Forecasting

The full experimental results provide detailed breakdowns for Monash benchmarking, probabilistic forecasting, and long-sequence forecasting, including alternative LLMTime evaluations and multiple forecasting metrics. Long-sequence results are organized by prediction length, with full-shot comparisons attributed to Liu et al. (2023b).

  • D.1. In-distribution Forecasting: Monash Time Series Forecasting Benchmark: Figure 7 extends the aggregate Monash benchmark results and distinguishes full-dataset aggregation from aggregation over the LLaMA2-covered subset.GPT3.5 denotes the authors’ reproduction based on the GPT3.5 API.
  • D.1. In-distribution Forecasting: Monash Time Series Forecasting Benchmark: Table 20 reports the full Monash benchmark results using MAE, including GPT3.5 and LLaMA2 LLMTime variants.GPT3.5 is reproduced using GPT3.5-Turbo-Instruct, while LLaMA2 results come from Gruver et al. (2023).
  • D.1. In-distribution Forecasting: Monash Time Series Forecasting Benchmark: The Monash comparison includes two LLMTime versions: GPT3.5, reproduced by running the original code, and LLaMA2, reported for only a subset of datasets.The reproduction uses GPT3.5-Turbo-Instruct because text-davinci-003 has been deprecated.
  • D.2. Out-of-distribution Forecasting: Probabilistic Forecasting: Table 21 gives full probabilistic forecasting results together with sMAPE, MASE, ND, and NRMSE point-forecasting metrics.Best results are bolded and second-best results are underlined.
  • D.2. Out-of-distribution Forecasting: Probabilistic Forecasting: The probabilistic experiments report both distributional forecasting outcomes and additional point-forecasting evaluation measures.The listed metrics are symmetric mean absolute percentage error, mean absolute scaled error, normalized deviation, and normalized root mean squared error.
  • D.3. Out-of-distribution Forecasting: Long Sequence Forecasting: Table 22 provides the complete long-sequence forecasting breakdown, listing results separately for each prediction length.Best results are highlighted in bold, and second-best results are underlined.
  • D.3. Out-of-distribution Forecasting: Long Sequence Forecasting: The long-sequence forecasting table presents full experimental results with best and second-best entries visually distinguished.The table’s comparison format highlights the strongest and next-strongest results.
  • D.3. Out-of-distribution Forecasting: Long Sequence Forecasting: Full-shot long-sequence forecasting results are sourced from Liu et al. (2023b).This attribution identifies the source of the full-shot comparison results.

D.4. Computation Costs · E. Forecast Visualizations

Moirai’s inference cost is comparable to similarly sized deep-learning models, while patch inputs and masked encoding reduce computation for multi-step forecasting; its zero-shot use can amortize pre-training costs across datasets. Forecast visualizations show zero-shot predictions from MoiraiBase on four benchmark datasets.

  • D.4. Computation Costs: Moirai’s inference cost is similar to other deep-learning models when model size and settings are held constant.The analysis varies context and prediction lengths.
  • D.4. Computation Costs: Patch-based inputs significantly reduce computation by decreasing the number of input tokens.
  • D.4. Computation Costs: The masked encoder predicts multiple forecast steps in one forward pass, unlike decoder-only Transformers and RNNs, which predict autoregressively.Autoregressive methods require multiple forward passes for multi-step forecasts, making long horizons costly.
  • D.4. Computation Costs: Zero-shot forecasting lets Moirai avoid repeated per-dataset training and hyperparameter tuning required by standard baselines.Moirai incurs increased costs from its model size, while standard approaches are retrained on each new dataset.
  • D.4. Computation Costs: As Moirai is applied to new datasets, its pre-training costs are amortized and become cheaper relative to repeatedly trained standard approaches.
  • E. Forecast Visualizations: Figure 8 visualizes zero-shot forecasts from MoiraiBase on the ETTh1 and ETTm1 datasets.
  • E. Forecast Visualizations: Figure 9 visualizes zero-shot forecasts from MoiraiBase on the Istanbul Traffic and Turkey Power datasets.
Loading 2402.02592v2…