PaperlayerView exact revision on arXiv

Source-linked AI summary

Moirai 2.0: When Less Is More for Time Series Forecasting

Chenghao Liu, Taha Aksu, Juncheng Liu, Xu Liu, Hanshu Yan, Quang Pham, Silvio Savarese, Doyen Sahoo, Caiming Xiong, Junnan Li

arXiv:2511.11698v3cs.LG

TL;DR

Time-series foundation models must generalize across domains with nonstationary, multiscale, irregular, and incomplete data. Moirai 2.0 addresses this with a decoder-only model trained on 36M series, quantile forecasting, and multi-token prediction; it achieves top-5 pretrained-model performance with favorable accuracy, speed, and size trade-offs. Performance degrades with larger parameter counts and longer horizons, defining key boundaries for future scaling.

  • Problem

    Time-series forecasting requires cross-domain generalization despite nonstationarity, multi-scale structure, irregular sampling, and incomplete observations.

  • Method

    Moirai 2.0 combines a decoder-only autoregressive architecture, quantile loss, multi-token prediction, and a new corpus of 36M time series.

  • Results

    Top 5 performance among pretrained models on GIFT-EVAL accompanies a favorable trade-off between accuracy, inference speed, and parameter count.

  • Takeaways & Limitations

    Decoder-only modeling and recursive multi-quantile decoding account for most of the observed gains in the reported ablations.

  • Takeaways & Limitations

    Performance degrades as parameter count increases and forecast horizons become longer, suggesting a mismatch between architecture and available data.

Abstract

from arXiv · show

We introduce Moirai 2.0, a decoder-only time-series foundation model trained on a new corpus of 36M series. The model adopts quantile forecasting and multi-token prediction, improving both probabilistic accuracy and inference efficiency. On the Gift-Eval benchmark, it ranks among the top pretrained models while achieving a strong trade-off between accuracy, speed, and model size. Compared to Moirai 1.0, Moirai 2.0 replaces masked-encoder training, multi-patch inputs, and mixture-distribution outputs with a simpler decoder-only architecture, single patch, and quantile loss. Ablation studies isolate these changes -- showing that the decoder-only backbone along with recursive multi-quantile decoding contribute most to the gains. Additional experiments show that Moirai 2.0 outperforms larger models from the same family and exhibits robust domain-level results. In terms of efficiency and model size, Moirai 2.0 is twice as fast and thirty times smaller than its prior best version, Moirai 1.0-Large, while also performing better. Model performance plateaus with increasing parameter count and declines at longer horizons, motivating future work on data scaling and long-horizon modeling. We release code and evaluation details to support further research.

1 Introduction

Time-series foundation models aim to generalize across diverse domains despite challenging temporal structure and incomplete observations. Moirai 2.0 simplifies Moirai 1.0’s design, trains on 36M series, and improves the accuracy–efficiency trade-off.

  • Time-series forecasting supports capacity planning, anomaly response, and risk management across many operational domains.
  • Nonstationarity, multi-scale structure, irregular sampling, and incomplete observations make cross-domain generalization challenging.
  • Moirai 1.0’s masked-encoder training, multi-patch inputs, and mixture-distribution outputs limited data utilization, frequency learning, and optimization efficiency.
  • 36M time series and ∼295B observations support Moirai 2.0’s pretraining, while its revised design improves accuracy and efficiency.
  • 5th among 37 foundation models on GIFT-EVAL, Moirai 2.0 offers a favorable trade-off between inference speed, model size, and accuracy.

2 Background & Related Work

Time-series forecasting spans statistical, deep-learning, and foundation-model approaches, with foundation models targeting reusable cross-domain prediction. Moirai 2.0 uses decoder-only quantile forecasting aligned with probabilistic evaluation.

  • Point forecasting predicts H future values from a historical context of c values, and multivariate forecasting predicts multiple variates simultaneously.
  • Moirai 2.0 treats multivariate forecasting as independent univariate tasks and predicts quantiles at each future time step.
  • Quantile levels are set from 0.1 through 0.9, providing a direct representation of predictive uncertainty.
  • Statistical models rely on local historical statistics, whereas deep-learning models are often trained separately for each dataset.
  • Time-series foundation models have expanded rapidly, with more than 25 models publicized in the last 2 years.
  • Moirai 2.0 combines a decoder-only architecture with quantile forecasts optimized by quantile loss, directly aligning outputs with CRPS.

3 Model Architecture and Training

Moirai 2.0 processes patched time series with a decoder-only Transformer and produces multi-token quantile forecasts. Its training and inference design targets probabilistic accuracy, efficiency, and reduced long-horizon error accumulation.

  • Architecture overview: Figure 1 presents the pipeline from patched time-series inputs through the Transformer backbone to predicted quantiles.
  • Input processing: Moirai 2.0 is designed for univariate inputs, representing multivariate data as independent univariate series.
  • Input processing: The input projection maps each patch from R2pin to Rd using a residual block with SiLU activation, weights W, and bias b.
  • Input processing: Normalization statistics use only the first 30% of the series, reserving the remaining 70% for causal pretraining to avoid future leakage.
  • Transformer backbone: Causal self-attention ensures each token depends only on preceding tokens and itself, preserving the decoder-only autoregressive property.
  • Output projection: Multi-token prediction generates forecasts for multiple future patches per output token, improving efficiency and reducing long-term error accumulation.

Loss Function

Moirai 2.0 uses quantile loss to train probabilistic forecasts across nine quantile levels and predicted horizons. The training objective averages these losses while excluding masked or missing targets, with optional weighting for selected quantiles.

  • Moirai 2.0 predicts 9 quantile levels, evenly spaced from 0.1 to 0.9.The quantile set is Q = {0.1, 0.2, ..., 0.9}.
  • Quantile loss, also called pinball loss, trains the model to learn the conditional distribution of future values.
  • The total training loss averages quantile losses across all levels and H time steps in K predicted patches.Masked or missing target values do not contribute to the loss.
  • All quantile levels receive equal weight by default, but selected quantiles can be emphasized with different weights.
  • Patch-level random masking masks 50% of input patches to encourage robust representation learning and improve handling of missing data segments.
  • Normalization uses only the initial 30% of each sample, so distribution shifts against the remaining 70% can destabilize training; anomalous samples are filtered.

Inference

Moirai 2.0 resolves the dimensional mismatch of autoregressive quantile forecasting through multi-quantile decoding. It expands several quantile-conditioned histories, decodes ahead, and collapses the results into a fixed quantile set.

  • Feeding all predicted quantiles back into the model creates a dimensional mismatch, while collapsing them to one value discards uncertainty information.
  • Autoregressive multi-quantile decoding expands each previous-step quantile path, decodes the next step, and collapses the expanded forecasts.The procedure is described conceptually as a depth-2 beam search.
  • The algorithm initializes by predicting the first-step quantiles directly from the context before beginning expansion.
  • At each subsequent step, the decoder forms m histories by appending each previous quantile prediction to the current context.
  • Each expanded history is decoded one step ahead to produce forecasts conditioned on quantile pairs.
  • In practice, Moirai 2.0 outputs multiple patch tokens per iteration rather than only the single-value pseudocode shown for clarity.

3.3 Discussion and Comparison on Moirai 2.0 and Moirai 1.0

Moirai 2.0 simplifies Moirai 1.0’s design with a decoder-only backbone, a single patch size, and quantile loss. These changes improve data and computational efficiency while supporting probabilistic forecasting and faster repeated inference.

  • Moirai 2.0 replaces Moirai 1.0’s masked encoder, multi-patch inputs, and distribution loss with decoder-only modeling, a single patch size, and quantile loss.
  • The decoder-only architecture computes T−1 losses across all tokens or patches, whereas Moirai 1.0’s 15% masking engages only 15% of tokens.This training strategy makes Moirai 2.0 more data-efficient.
  • KV caching can speed repeated forecasting by reusing key–value representations instead of recomputing them at every query.The benefit is relevant when users extend forecasts incrementally or request predictions repeatedly.
  • A single patch size improves computational efficiency and forecasting accuracy while simplifying training and inference implementation.
  • Moirai 2.0 trains with quantile loss and generates |Q| quantiles, replacing Moirai 1.0’s mixture-distribution sampling and distribution NLL loss.Quantile loss can directly optimize operational quantiles such as q = 0.9 for capacity planning.

4 Pretraining Datasets

Moirai 2.0 is pretrained on a diverse corpus combining benchmark data, synthetic series, and internal telemetry. The corpus contains 36M time series and approximately 295B observations, while benchmark results place the model among leading pretrained systems.

  • 36M time series and approximately 295B observations form Moirai 2.0’s new pretraining corpus.The corpus combines five complementary sources.
  • The corpus includes non-leaking GIFT-EVAL PRETRAIN data and the GIFT-EVAL TRAINTEST train split.GIFT-EVAL PRETRAIN contains 3.25M series and 230B observations under the paper’s counting convention.
  • Chronos-Mixup and KernelSynth add synthetic time series generated through convex combinations and Gaussian-process kernel constructions.KernelSynth contributes 1M series and 1.02B observations.
  • Internal Salesforce CloudOps telemetry contributes approximately 2.15M univariate daily series totaling roughly 1.48B observations from about one year.Series with substantial missing values were filtered before training.
  • On GIFT-EVAL, Moirai 2.0 ranks among the top models under normalized MASE and normalized CRPS, where lower scores are better.The figure excludes ensemble methods and models without reproducible code.

5 Evaluation

Moirai 2.0 is evaluated on GIFT-Eval across overall accuracy, domains, horizons, efficiency, scaling, and architectural ablations. It ranks highly while offering substantially improved efficiency, although performance weakens for longer horizons and does not improve with larger variants.

  • 5.1 Results on GIFT-Eval Benchmark: GIFT-Eval evaluation compares Moirai 2.0 with pretrained foundation models across 97 task configurations and 55 datasets, using normalized MASE and CRPS.The study filters out agentic solutions, fine-tuned models, and models without reproducible code, leaving 30 foundation models.
  • 5.1 Results on GIFT-Eval Benchmark: 5th on MASE and 6th on CRPS, Moirai 2.0 substantially outperforms Moirai-Large despite having fewer parameters.Scores are divided by the seasonal-naive baseline and aggregated using the geometric mean across tasks.
  • Fine Grained Results: Moirai 2.0 reaches the top-10 across most domains, but Nature tasks remain a notable exception and Transport is the only domain where Moirai-Large stays highly competitive.The authors suggest that underrepresentation of natural and environmental series in pretraining may contribute to the Nature gap.
  • Fine Grained Results: 4th, 6th, and 8th are Moirai 2.0’s ranks for short, medium, and long prediction lengths, respectively, showing declining relative advantage as horizons grow.Figure 4 orders the top-10 foundation models by MASE rank, where lower is better.
  • 5.2 Efficiency Comparison: 30× smaller and 2× faster than Moirai-Large, Moirai 2.0 also achieves significantly better accuracy, while Granite-FlowState-R1 is more accurate but roughly three times slower.The efficiency comparison uses 12 representative GIFT-Eval tasks timed on a single H200 GPU with official replication code.
  • 5.3 Scaling Experiments: 8× and 30× parameter increases in the base and large variants do not improve GIFT-Eval MASE or CRPS over the small model.The results suggest that further gains may require more pretraining data, architectural innovations, or targeted regularization rather than parameter growth alone.
  • 5.4 Ablation Study: Quantile loss produces the largest single ablation improvement, while recursive decoding further reduces error and multi-token prediction yields the best CRPS with similar MASE.The final variant adds a residual projection block and achieves the best overall performance.

6 Limitations and Future Work

Moirai 2.0 drops multivariate forecasting and covariates, while its scaling behavior remains constrained by the available data and longer forecast horizons. Future work considers data-aligned scaling, long-horizon architectures, agentic solutions, and multimodal reasoning.

  • Limitations: Moirai 2.0 drops support for multivariate forecasting and covariates because the authors observed minimal benefit and limited high-quality datasets.The authors may revisit these capabilities if suitable data become available, potentially through synthetic data generation.
  • Limitations: Increasing parameter count degrades performance, suggesting a mismatch between the model architecture and available data.The authors propose aligning data scaling with model capacity to address this challenge.
  • Limitations: Performance weakens as forecast horizons lengthen, motivating architectural improvements for long-horizon forecasting.The authors identify long-horizon performance as a future research direction.
  • Future Work: Agentic systems combining time-series analysis with large-language-model reasoning are proposed as a promising future direction.Recent agent-based submissions to GIFT-Eval are cited as evidence of this paradigm’s potential.
  • Future Work: Multimodal foundation models integrating text, images, and time series are proposed to enrich forecasting with contextual information.The paper notes that early work exists but that substantial questions remain open.

7 Conclusion

Moirai 2.0 combines decoder-only modeling, quantile forecasting, and multi-token prediction with a 36-million-series pretraining corpus. It achieves top-five pretrained-model performance on GIFT-EVAL with a favorable accuracy–speed–parameter trade-off, while scaling and long-horizon limitations remain.

  • Conclusion: Moirai 2.0 pairs a decoder-only architecture with quantile forecasting and multi-token prediction, trained on 36 million time series.The model uses a newly curated diverse pretraining dataset.
  • Conclusion: Moirai 2.0 achieves top-five performance among pretrained models on GIFT-EVAL while balancing accuracy, inference speed, and parameter count.The conclusion presents this as the model’s principal benchmark outcome.
  • Conclusion: Decoder-only architecture and recursive multi-quantile decoding account for most of the observed gains in the ablation studies.The ablations isolate the contribution of these design choices.
  • Conclusion: Diminishing returns from parameter scaling and degraded performance at longer horizons remain limitations requiring data and architectural improvements.The paper proposes data scaling aligned with model capacity and long-horizon architectural work.

A Appendix

The appendix breaks down GIFT-EVAL leaderboard rankings by variate type, frequency, trendiness, seasonality, entropy, Hurst value, stability, and lumpiness. Each breakdown displays the top ten foundation models ordered by MASE rank, where lower is better.

  • GIFT-EVAL breakdowns: Leaderboard results are split by variate type and frequency, with top-10 foundation models ordered by MASE rank.The variate-type comparison covers both univariate and multivariate series; frequency results are grouped by frequency type.
  • Series characteristics: Leaderboard results are broken down by trendiness and seasonal strength, with higher values indicating stronger trends or seasonal patterns.Models are displayed in MASE-rank order, with lower ranks better.
  • Series characteristics: Leaderboard results are also grouped by entropy, Hurst value, stability, and lumpiness to characterize forecastability, persistence, stability, and variance variability.Entropy, Hurst, stability, and lumpiness each describe a different time-series property used for subgroup analysis.
Loading 2511.11698v3…