Source-linked AI summary
Moirai-MoE: Empowering Time Series Foundation Models with Sparse Mixture of Experts
Xu Liu, Juncheng Liu, Gerald Woo, Taha Aksu, Yuxuan Liang, Roger Zimmermann, Chenghao Liu, Silvio Savarese, Caiming Xiong, Doyen Sahoo
TL;DR
Time series foundation models face the challenge of unified training across heterogeneous data, while frequency-level specialization can miss pattern diversity and within-series distribution changes. MOIRAI-MOE uses sparse Transformer experts for data-driven token-level specialization, and experiments report improved performance over existing foundation models in in-distribution and zero-shot settings.
Problem
Frequency-level specialization can miss similar patterns across frequencies, diverse patterns within a frequency, and distribution changes within a single time series.
Source: §1 · p. 2Method
MOIRAI-MOE uses one input/output projection layer and sparse Transformer experts to model diverse patterns through data-driven token-level specialization.
Source: §1 · p. 2Results
Across 39 datasets, MOIRAI-MOE outperforms existing foundation models in in-distribution and zero-shot forecasting, with up to 17% improvement over MOIRAI at the same model size and up to 65× fewer activated parameters than other foundation models.
Source: §1 · pp. 2–3Takeaways & Limitations
The model analyses report frequency-invariant representations and progressive denoising through the model, offering insights into time series MoE foundation models.
Source: §1 · pp. 2–3
Abstract
from arXiv · showhide
Time series foundation models have demonstrated impressive performance as zero-shot forecasters. However, achieving effectively unified training on time series remains an open challenge. Existing approaches introduce some level of model specialization to account for the highly heterogeneous nature of time series data. For instance, Moirai pursues unified training by employing multiple input/output projection layers, each tailored to handle time series at a specific frequency. Similarly, TimesFM maintains a frequency embedding dictionary for this purpose. We identify two major drawbacks to this human-imposed frequency-level model specialization: (1) Frequency is not a reliable indicator of the underlying patterns in time series. For example, time series with different frequencies can display similar patterns, while those with the same frequency may exhibit varied patterns. (2) Non-stationarity is an inherent property of real-world time series, leading to varied distributions even within a short context window of a single time series. Frequency-level specialization is too coarse-grained to capture this level of diversity. To address these limitations, this paper introduces Moirai-MoE, using a single input/output projection layer while delegating the modeling of diverse time series patterns to the sparse mixture of experts (MoE) within Transformers. With these designs, Moirai-MoE reduces reliance on human-defined heuristics and enables automatic token-level specialization. Extensive experiments on 39 datasets demonstrate the superiority of Moirai-MoE over existing foundation models in both in-distribution and zero-shot scenarios. Furthermore, this study conducts comprehensive model analyses to explore the inner workings of time series MoE foundation models and provides valuable insights for future research.
1 INTRODUCTION
Frequency-level specialization struggles to capture the diversity and non-stationarity of time series patterns. MOIRAI-MOE instead uses sparse experts for data-driven token-level specialization, and the paper reports improvements over existing foundation models alongside analyses of how the model works.
1 INTRODUCTION
- Frequency is an imperfect proxy for time series patterns: different frequencies can share similar patterns, while the same frequency can contain diverse patterns and distributions can vary within one series.The paper argues frequency-level specialization is too coarse-grained for this diversity.Source: §1 · p. 2
- MOIRAI-MOE uses a single input/output projection layer and sparse Transformer experts to model diverse patterns through data-driven token-level specialization.It also introduces a gating function guided by cluster centroids from a pretrained model.Source: §1 · p. 2
- Experiments across 39 datasets report that MOIRAI-MOE surpasses TimesFM, Chronos, and MOIRAI in in-distribution and zero-shot forecasting.The paper also reports analyses finding frequency-invariant representations and progressive denoising through the model.Source: §1 · p. 2
- MOIRAI-MOE delivers up to 17% performance improvement over MOIRAI at the same model size and outperforms other time series foundation models with up to 65× fewer activated parameters.The introduction reports these comparisons across experiments on 39 datasets.Source: §1 · p. 3
- The authors conduct model analyses to deepen understanding of time series MoE foundation models and derive insights for future research.The analyses are presented as a contribution alongside the forecasting evaluation.Source: §1 · pp. 2–3
2 RELATED WORK
Prior time series foundation models address heterogeneity through dataset-level or frequency-level specialization, or use a shared architecture without specialization. This paper positions sparse MoE as a way to specialize automatically at the token level.
2 RELATED WORK
- TEMPO and UniTime use language prompts for dataset-level specialization, while MOIRAI uses frequency-specific projections and TimesFM uses a frequency embedding dictionary [Cao et al., 2024; Liu et al., 2024a; Woo et al., 2024; Das et al., 2024].These approaches use different data identifiers to distinguish time series.Source: §2 · p. 3
- Chronos, Lag-LLaMA, Moment, and Timer use the same architecture across time series, which the paper says can increase learning complexity and parameter demands [Ansari et al., 2024; Rasul et al., 2023; Goswami et al., 2024; Liu et al., 2024c].The passage frames this as a potential cost of not incorporating specialization modules.Source: §2 · p. 3
- The paper proposes sparse MoE for token-level specialization, with different tokens handled by specialized experts and similar tokens sharing parameter space.MoE layers replace Transformer feed-forward networks with experts and a gating function; assignments are usually sparse for computational efficiency [Jiang et al., 2024].Source: §2 · p. 3
3 METHODOLOGY
MOIRAI-MOE replaces frequency-specific projections with a single projection and sparse Transformer experts, using pretrained-model knowledge for gating. Its decoder-only objective supports parallel learning across context lengths.
3.1 TIME SERIES TOKEN CONSTRUCTION
- Adjacent values are grouped into non-overlapping patches, normalized with a masking ratio that reserves part of the sequence for robust normalization, then projected into time-series tokens.The reserved portion does not contribute to prediction loss, and the projection is a residual MLP.Source: §3.1 · pp. 3–4
3.2 MIXTURE OF EXPERTS FOR TRANSFORMERS
- The decoder-only Transformer flattens variates into a sequence for causal attention, then replaces each feed-forward network with sparse experts, activating K = 2 experts per token.Tokens can attend to preceding tokens from other variates.Source: §3.2 · p. 4
3.2.1 GATING FUNCTION
- Sparse gating can create load imbalance, so an auxiliary loss encourages even token distribution across experts using each expert’s routed-token fraction and gating-probability share.The loss is defined for a batch of tokens.Source: §3.2.1 · p. 5
- The proposed gate clusters pretrained MOIRAI token representations and uses each token’s Euclidean distances to cluster centroids as expert-affinity scores.The number of clusters matches the number of experts.Source: §3.2.1 · p. 5
3.3 TRAINING OBJECTIVE
- Training minimizes negative log-likelihood for the predictive distribution of the next token, whose mixture parameters come from Transformer outputs followed by a single output projection.Source: §3.3 · p. 5
4 EXPERIMENTS
Experiments evaluate MOIRAI-MOE on in-distribution and zero-shot forecasting, then examine its design choices, expert behavior, and inference efficiency. The reported results favor MOIRAI-MOE over comparison foundation models across both evaluation settings.
4.1 MOIRAI-MOE SETUP
- With K = 2 of 32 experts, MOIRAI-MOES small/base models use 11M/86M activated parameters and train for 50,000/250,000 LOTSA steps.The patch size is 16 and the decoder-only masking ratio is 0.3.Source: §4.1 · pp. 5–6
4.2 MAIN RESULTS
- 17% lower performance error than MOIRAIS, 8% than MOIRAIB, and 7% than MOIRAIL are reported for MOIRAI-MOES on the 29-dataset in-distribution evaluation.MOIRAI-MOEB improves a further 3% over MOIRAI-MOES.Source: Figure 3 · p. 6, §4.2 · p. 7
- 3%–14% CRPS and 8%–16% MASE improvements over all MOIRAI sizes are reported for MOIRAI-MOES on zero-shot datasets; MOIRAI-MOEB achieves the best zero-shot performance.MOIRAI-MOES has 11M activated parameters, 28× fewer than MOIRAIL.Source: Table 2 · p. 6, §4.2 · p. 7
4.3 ABLATION STUDIES
- Objective-only changes yield modest gains; decoder-only training beats masked encoding at each tested step, while token clustering outperforms alternative gates as expert count increases.Decoder-only training at 50k steps matches masked-encoder training at 100k steps.Source: §4.3 · pp. 7–8, Table 3 · p. 7, Figure 4 · p. 8
4.4 MODEL ANALYSES
- 25% MAE improvement on NN5 Daily (5.37 to 4.04), 35% on Traffic Hourly (0.02 to 0.013), and 4% on Covid Daily Deaths (124.32 to 119) accompany data-driven embeddings.MOIRAI-MOE blends embeddings of similar-pattern datasets across frequencies and separates distinct-pattern datasets at the same frequency.Source: §4.4 · p. 8, Figure 5 · p. 8
- Expert allocation differs by frequency in shallow layers but is nearly identical by layer 6, indicating frequency-invariant high-level representations.As tokens are aggregated, experts become more concentrated in deeper layers.Source: §4.4 · p. 9, Figure 6 · p. 9
- Only 3 of 32 experts are selected in the final layer; the authors interpret this convergence as progressive denoising through the model.Some experts are rarely selected; pruning them for compression is left for future work.Source: §4.4 · p. 9, Figure 6 · p. 9
- In shallow layers, expert choices for hourly data follow periodic patterns corresponding to phases such as rising, peaks, and falling.The analysis uses hourly series with context lengths of at least 1,000.Source: §4.4 · p. 10, Figure 7 · p. 10
5 CONCLUSION
MOIRAI-MOE uses sparse experts for data-driven modeling of diverse time-series patterns and enables automatic token-level specialization.
5 CONCLUSION
- MOIRAI-MOE uses sparse experts to model diverse time-series patterns in a data-driven manner, improving performance over MOIRAI, TimesFM, and Chronos with fewer activated parameters.The authors also report comprehensive analyses of time-series MoE foundation models.Source: §5 · p. 11
A EXPERIMENTAL DETAILS
The experiments cover in-distribution and zero-shot forecasting datasets and describe evaluation procedures, comparison models, and full-shot baseline tuning.
A EXPERIMENTAL DETAILS
- The in-distribution evaluation uses 29 Monash benchmark datasets, with dataset statistics and full foundation-model MAE results summarized in Tables 5 and 6.The zero-shot datasets span five domains and frequencies from minute-level to weekly.Source: A EXPERIMENTAL DETAILS · pp. 14–15, Table 5 · p. 14, Table 6 · p. 14
- Zero-shot evaluation uses non-overlapping rolling windows with stride equal to the prediction length, and reserves the final h*r time steps for testing.Validation is the forecast horizon immediately before the test set; training uses all preceding data.Source: A EXPERIMENTAL DETAILS · p. 15
- The comparison models include TiDE, PatchTST, and iTransformer, alongside LLMTime, TimesFM, Chronos, and MOIRAI.The supplied descriptions characterize their architectures and forecasting approaches.Source: A EXPERIMENTAL DETAILS · p. 15
- MOIRAI is described as trained on LOTSA, a dataset containing over 27 billion observations across nine diverse domains.The method overview also identifies MOIRAI-MOE as capable of automatic token-level specialization.Source: A EXPERIMENTAL DETAILS · p. 15
- The three full-shot baselines were tuned with random search over 15 training runs, and results use the best validation CRPS.Search included learning rates from 1e-6 to 1e-3 and context length l = m*h with m in [2, 20].Source: Table 8 · p. 15, A EXPERIMENTAL DETAILS · p. 16
B ADDITIONAL RESULTS
Figure 8 examines effects of patch size and masking ratio for MOIRAI-MOES.
B ADDITIONAL RESULTS
- Figure 8 examines the effects of patch size and masking ratio for MOIRAI-MOES.Source: Figure 8 · p. 16
B.1 EFFECTS OF PATCH SIZE
Patch size 16 provides the best Monash performance among the tested choices while maintaining reasonable inference speed.
B.1 EFFECTS OF PATCH SIZE
- Patch size 16 yields the best Monash performance; smaller patches increase token count, GPU memory use, and inference time.Patch size 4 can take over a day to complete all evaluations.Source: §B.1 · p. 16
B.2 EFFECTS OF MASKING RATIO
The section examines how the masking ratio affects robust normalization and reports 0.3 as the best-performing ratio.
B.2 EFFECTS OF MASKING RATIO
- The masking ratio r controls how much of the sequence is used solely to calculate a robust normalizer and help mitigate distribution shift.The selected data range should be small enough to avoid excessive masking but representative enough for normalization.Source: §B.2 · p. 16
- A masking ratio of 0.3 delivers the best performance on the Monash benchmark.Source: §B.2 · p. 16
- A ratio of 0.1 may leave too little data for robust normalization, while 0.5 may hinder parallel learning efficiency in decoder-only training.Source: §B.2 · p. 16
B.3 EXPERT DISTRIBUTIONS OF DIFFERENT GATING FUNCTION
On the Monash benchmark, token clustering ranks highest among the three gating functions, despite producing the least diverse expert selections.
B.3 EXPERT DISTRIBUTIONS OF DIFFERENT GATING FUNCTION
- Token clustering uses centroids from pretrained MOIRAI token representations to assign experts by Euclidean distance, and outperforms randomly learned gating from scratch.Centroids are updated by mini-batch k-means on self-attention outputs from LOTSA data corresponding to 100 epochs; cluster count matches the number of experts.Source: §B.3 · pp. 16–17
- Performance ranks Token Clusters > Linear Projection with Load Balancing > Pure Linear Projection, while allocation diversity ranks in the reverse order.Figure 9 shows expert allocation distributions aggregated across Monash datasets.Source: §B.3 · p. 17, Figure 9 · p. 17
- The authors attribute lower diversity under clustering to targeted routing, while load balancing spreads expert use and pure projection lacks either mechanism.They suggest targeted routing aligns experts with structured patterns; load balancing may improve generalization and performance versus pure projection, but may be less targeted than clustering.Source: §B.3 · p. 17
C VISUALIZATION
The section visualizes three datasets used in the model analyses to help readers understand their time-series patterns.
C VISUALIZATION
- Figure 10 visualizes NN5 Daily data, including context length and forecast results.Source: Figure 10 · p. 18
- Figure 11 visualizes Traffic Hourly data, including context length and forecast results.Source: Figure 11 · p. 19
- Figure 12 visualizes Covid Daily Deaths data, including context length and forecast results.Source: Figure 12 · p. 20