PaperlayerView exact revision on arXiv

Source-linked AI summary

Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Experts

Xiaoming Shi, Shiyu Wang, Yuqi Nie, Dianqi Li, Zhou Ye, Qingsong Wen, Ming Jin

arXiv:2409.16040v4cs.LGcs.AI

TL;DR

Time-series foundation models have remained limited in scale and costly to operate, leaving open how to achieve universal forecasting with greater capability and efficiency. TIME-MOE uses a sparse mixture-of-experts decoder-only architecture, trained on Time-300B with flexible multi-resolution forecasting. Scaled to 2.4 billion parameters, it improves forecasting and consistently outperforms dense models under comparable computational budgets.

  • Problem

    Time-series foundation models remain limited in scale, flexibility, and general-purpose forecasting capability despite the success of large-scale pre-training in other domains.

  • Method

    TIME-MOE is a sparsely activated decoder-only transformer trained on Time-300B, using mixture-of-experts layers and multi-resolution forecasting heads for flexible horizons and context lengths.

  • Results

    20% and 24% average forecasting-error reductions are reported for zero-shot and in-distribution scenarios, while the 2.4-billion-parameter model consistently outperforms dense models with equivalent computational budgets.

  • Takeaways & Limitations

    The results support scaling laws for time-series forecasting and demonstrate that sparse models can increase capability without proportional inference-cost growth.

Abstract

from arXiv · show

Deep learning for time series forecasting has seen significant advancements over the past decades. However, despite the success of large-scale pre-training in language and vision domains, pre-trained time series models remain limited in scale and operate at a high cost, hindering the development of larger capable forecasting models in real-world applications. In response, we introduce Time-MoE, a scalable and unified architecture designed to pre-train larger, more capable forecasting foundation models while reducing inference costs. By leveraging a sparse mixture-of-experts (MoE) design, Time-MoE enhances computational efficiency by activating only a subset of networks for each prediction, reducing computational load while maintaining high model capacity. This allows Time-MoE to scale effectively without a corresponding increase in inference costs. Time-MoE comprises a family of decoder-only transformer models that operate in an auto-regressive manner and support flexible forecasting horizons with varying input context lengths. We pre-trained these models on our newly introduced large-scale data Time-300B, which spans over 9 domains and encompassing over 300 billion time points. For the first time, we scaled a time series foundation model up to 2.4 billion parameters, achieving significantly improved forecasting precision. Our results validate the applicability of scaling laws for training tokens and model size in the context of time series forecasting. Compared to dense models with the same number of activated parameters or equivalent computation budgets, our models consistently outperform them by large margin. These advancements position Time-MoE as a state-of-the-art solution for tackling real-world time series forecasting challenges with superior capability, efficiency, and flexibility.

1 INTRODUCTION

Existing time-series foundation models remain limited in scale, flexibility, and general-purpose forecasting capability. TIME-MOE addresses these challenges with a scalable sparse architecture and large-scale pre-training data.

  • Motivation: Existing time-series foundation models are generally small and have limited task-solving capabilities compared with domain-specific models.The field has only recently begun developing unified, general-purpose forecasting models.
  • Motivation: Scaling laws are established in language and vision but have not been thoroughly investigated for time-series forecasting.This motivates studying how model size and training tokens affect forecasting performance.
  • Motivation: Most time-series foundation models struggle to support arbitrary context lengths, forecasting horizons, input variables, frequencies, and distributions simultaneously.Examples include Timer’s limited arbitrary output lengths, Moment’s fixed context length, and Moirai’s hardcoded input and output heuristics.
  • TIME-MOE: TIME-MOE combines decoder-only transformers with mixture-of-experts to scale forecasting models while reducing computational costs.Its autoregressive design supports any forecasting horizon and context lengths up to 4096.
  • TIME-MOE: TIME-MOE activates only a subset of networks per prediction, preserving high model capacity without significantly increasing inference costs.The models are trained on data spanning 9 domains and over 300 billion time points, with multi-resolution forecasting heads.
  • Contributions: 20% and 24% average forecasting-error reductions are reported in zero-shot and in-distribution scenarios, respectively, across six real-world benchmarks.The comparison is against other time-series foundation models with a similar number of activated parameters.

2 RELATED WORK

Prior work includes task-specific univariate and multivariate forecasters, self-supervised time-series representation learning, and emerging large time-series foundation models. Sparse architectures offer an efficiency-oriented alternative, while TIME-MOE applies them to universal forecasting.

  • Time Series Forecasting: Time-series forecasting models include univariate approaches focused on individual series and multivariate approaches that model multiple variables.The related work categorizes deep forecasting methods into these two broad types.
  • Large Time Series Models: Self-supervised time-series methods commonly use masked reconstruction or contrastive learning but remain limited in data and model scale.Many such methods focus on in-domain learning and transfer.
  • Sparse Deep Learning for Time Series: Sparse networks dynamically route inputs to specialized expert networks and can provide comparable or superior generalization to dense models with greater efficiency.Mixture-of-experts is presented as a representative sparse architecture.
  • TIME-MOE: TIME-MOE uses a mixture-of-experts decoder-only transformer with input token embedding, MoE transformer blocks, and multi-resolution forecasting.The architecture is scaled to 2.4 billion parameters for universal forecasting.
  • Sparse Deep Learning for Time Series: MoLE and IME are distinguished from sparse models because their inputs pass through all heads before predictions are combined.This contrasts with selective expert routing in sparse architectures.

3 METHODOLOGY

TIME-MOE combines a sparsely activated decoder-only transformer with multi-resolution forecasting and a large-scale data-cleaning and training pipeline. Its design supports flexible forecasting while controlling computation and addressing optimization challenges in large MoE models.

  • Architecture: TIME-MOE uses a mixture-of-experts decoder-only transformer with input token embedding, MoE transformer blocks, and multi-resolution forecasting.The model is sparsely activated and scaled to 2.4 billion parameters while targeting improved zero-shot performance at the same computation.
  • Forecasting formulation: The model forecasts flexibly because context length and forecasting horizon can vary during inference, while channel independence supports any-variate inputs.The forecasting objective maps historical observations to future values, with T and H remaining flexible at inference.
  • Input representation: Point-wise tokenization preserves temporal information for variable-length sequences before SwiGLU embeds each time-series point.The embedding uses learnable parameters W and V with hidden dimension D.
  • MoE transformer block: The MoE block uses causal self-attention and expert networks, including one shared expert that consolidates common knowledge across contexts.Experts can receive individual points through single- or multi-expert routing, while RMSNorm and rotary positional embeddings support stable and flexible sequence processing.
  • Forecasting head: Multi-resolution forecasting uses separate output projections for different horizons, aggregates their errors during training, and combines predictions through greedy scheduling and ensemble learning.This design supports arbitrary horizons and improves prediction robustness during inference.
  • Data and optimization: The training pipeline cleans large-scale data, uses Huber loss for robustness to outliers, and adds an auxiliary balancing loss to reduce routing collapse.Time-MoE is trained on Time-300B, while the model family includes configurations up to 2.4 billion parameters and around 1 billion activated parameters.

4 MAIN RESULTS

TIME-MOE delivers strong zero-shot and fine-tuned forecasting results while scaling model capacity through sparse experts. Its evaluations also show computational savings, benefits from scaling data and parameters, and sensitivity to architectural choices.

  • Zero-shot Forecasting: TIME-MOE consistently achieves state-of-the-art zero-shot forecasting, with average MSE reductions exceeding 20% over competitive baselines.Against Chronoslarge, Moment, and Moirailarge, it reports average MSE reductions of 23%, 30%, and 11%, respectively.
  • Fine-tuning: 24% average MSE reduction is achieved after fine-tuning, while one-epoch downstream training substantially improves predictive performance.Performance continues to improve as TIME-MOE model size increases.
  • Ablation Study: 0.262 to 0.272 average MSE worsens when mixture-of-experts layers are replaced with standard feedforward networks.The ablation evaluates horizon-96 forecasting across six benchmarks.
  • Scalability Analysis: 78% lower training cost and 39% lower inference cost are reported for sparse models versus dense variants with equivalent activated parameters.The dense and sparse models use identical training setups and data.
  • Scalability Analysis: Larger datasets and model parameter counts consistently improve forecasting accuracy, with sparse models outperforming dense counterparts at matched scales.This pattern is shown using models trained from scratch across varying data sizes.
  • Training Precision: 12% faster training and 20% lower memory consumption are obtained with bfloat16 than with float32, while forecasting performance remains comparable.With flash-attention, training and inference speed improve by an additional 23% and 19%, respectively.

5 CONCLUSION

TIME-MOE is presented as a scalable, unified time series foundation model that combines sparse mixture-of-experts computation with large-scale pre-training. The paper concludes that scaling model size and data improves forecasting while retaining efficiency across zero-shot and fine-tuned settings.

  • Conclusion: TIME-MOE scales to 2.4 billion parameters with 1.1 billion activated, improving forecasting capability without compromising computational efficiency.The model consistently outperforms dense models with equivalent computational budgets.
  • Conclusion: Time-300B contains 309.09 billion time points across diverse sampling frequencies and supports the model’s large-scale pre-training.The dataset is organized for memory-efficient loading through binary files and metadata.
  • Conclusion: TIME-MOE is evaluated as a state-of-the-art solution for universal forecasting across zero-shot and fine-tuned scenarios.The comparison includes widely accepted benchmarks and settings.

B IMPLEMENTATION DETAILS

The implementation combines sparse expert routing, multi-resolution forecasting heads, and autoregressive scheduling. Training and evaluation use standardized transformer optimization, benchmark protocols, and MSE/MAE metrics.

  • Training Configuration: Models train for 100,000 steps with batch size 1,024 and maximum sequence length 4,096, using AdamW and horizons {1, 8, 32, 64}.Each iteration processes 4 million time points, with auxiliary loss factor α = 0.02.
  • Benchmark Details: Evaluation covers eight established datasets using MSE and MAE as the forecasting metrics.The benchmark suite includes Weather, Global Temp, and four ETT datasets.
  • Model Architecture: The MoE layer combines one shared expert with isolated experts, retaining only the top-k isolated-expert gating scores for each token.Shared-expert weights use Sigmoid normalization, while isolated-expert weights use Softmax normalization.
  • Model Architecture: An auxiliary loss balances expert load by penalizing experts with high gating scores and helps prevent routing collapse.The objective discourages stronger experts from receiving progressively more tokens than weaker experts.
  • Multi-resolution Forecasting: The multi-resolution head uses projections for distinct horizons, with each projection forecasting its corresponding number of future time steps.The horizon-specific projection maps the final hidden representation to a forecast of length p_j.
  • Multi-resolution Forecasting: Greedy scheduling selects the largest feasible projection not exceeding the remaining target duration, enabling arbitrary output lengths.The procedure repeats autoregressively until the requested horizon is covered.

C PROCESSED DATA ARCHIVE

Time-300B is constructed from heterogeneous time series through targeted quality filtering and missing-value handling. The resulting archive spans many domains, frequencies, datasets, and over 309 billion time points.

  • Data Quality: The cleaning pipeline measures missing-value and invalid-observation ratios to identify unstable or artificially imputed sequences.Missing values include nan and inf; invalid observations include excessive zeros in first- or second-order differences.
  • Data Quality: Missing-value processing splits sequences at nan or inf positions and retains sufficiently long contiguous subsequences instead of mean-imputing them.This approach is intended to preserve the original time-series pattern.
  • Data Quality: Invalid-observation processing scans fixed-length windows and discards windows whose quality ratios exceed a preset threshold.The described threshold is 0.2, based on zero and difference statistics.
  • Archive Construction: Time-300B contains 309.09 billion time points spanning sampling frequencies from seconds to yearly intervals.Datasets are split into binary files with metadata describing sequence positions for memory-efficient loading.
  • Dataset Coverage: The archive includes datasets from domains such as finance, healthcare, nature, sales, transport, and web data.The dataset tables record domain, sampling frequency, number of series, observations, and source.

D.1 ABLATION STUDY

Ablations show that TIME-MOE’s mixture-of-experts, multi-task optimization, and multi-resolution forecasting components each contribute to performance, with the complete multi-resolution configuration also improving inference speed.

  • Mixture-of-Experts: 0.262 to 0.272: Removing mixture-of-experts worsened average MSE across the ablation benchmarks.Dynamic routing enables more specialized processing of diverse input patterns.
  • Multi-Task Optimization: Removing multi-task optimization with only the horizon-32 forecasting head made TIME-MOEbase slightly inferior to the complete model.The ablation retained only the horizon-32 forecasting head and excluded multi-task optimization.
  • Multi-Resolution Forecasting: Four multi-resolution heads with receptive horizons 1, 8, 32, and 64 delivered optimal predictive performance and inference speed.Reducing the number of heads consistently decreased performance and increased inference time.
  • Overall Findings: The ablations identify mixture-of-experts, multi-task optimization, and multi-resolution forecasting as key architectural components for universal forecasting performance.The conclusion is stated across the reported component studies.

D.2 TRAINING PRECISION ANALYSIS

Training with bfloat16 preserves forecasting performance while improving computational efficiency over float32, especially when combined with flash-attention.

  • Predictive Performance: The BF16 and FP32 models have remarkably comparable forecasting performance.The comparison used identical configurations except for numerical precision.
  • Efficiency: 12%: bfloat16 improved training speed compared with float32.This reduces training time for large-scale models and supports faster experimentation.
  • Efficiency: 20%: bfloat16 reduced memory usage compared with float32.The lower memory footprint supports scaling models or deployment on memory-constrained hardware.
  • Flash-Attention: 23% additional training speed and 19% inference speed: bfloat16 combined with flash-attention produced further gains.FA denotes flash-attention in the efficiency comparison.
  • Implications: Bfloat16 and flash-attention provide a balance between model performance, computational efficiency, and resource utilization.The reported optimizations enable scalable deployment without sacrificing predictive accuracy.

D.3.1 TAXIBJ DATASET

The TaxiBJ evaluation uses a selected Beijing taxi in-flow period for short-term zero-shot forecasting. Across the reported benchmark comparisons and visual evaluations, TIME-MOE consistently outperforms other baselines and produces more accurate forecasts across multiple datasets.

  • Dataset: TaxiBJ uses Beijing taxicab in-flow data from November 1, 2015, to April 10, 2016, for short-term forecasting evaluation.The original dataset combines taxicab GPS and meteorological information across four collection intervals.
  • Evaluation Setup: All zero-shot models were evaluated with context length 512, and TaxiBJ results are summarized in Table 14.For MSE and MAE, lower values indicate better prediction; red marks the best and blue the second best.
  • Results: TIME-MOE models consistently outperform other baselines in short-term forecasting on TaxiBJ.The comparison includes the additional zero-shot and in-domain baseline results reported in Tables 15 and 16.
  • Visual Comparisons: Across ETTh1, ETTh2, ETTm1, ETTm2, Weather, and Global Temp, TIME-MOE produces more accurate forecasts than other foundation models.Figures 6–11 compare model predictions with ground truth at forecast horizon 96.
Loading 2409.16040v4…