Source-linked AI summary
Timer-S1: A Billion-Scale Time Series Foundation Model with Serial Scaling
Yong Liu, Xingjian Su, Shiyu Wang, Haoran Zhang, Haixuan Liu, Yuxuan Wang, Zhou Ye, Yang Xiang, Jianmin Wang, Mingsheng Long
TL;DR
Time series foundation models face a scaling bottleneck because forecasting is serial, while autoregressive rolling adds overhead and error accumulation. Timer-S1 combines sparse MoE and Serial-Token Prediction with trillion-point augmented pre-training and post-training, achieving state-of-the-art GIFT-Eval performance. The authors report best pre-trained-model MASE and CRPS scores and identify missing exogenous-covariate support as a limitation.
Problem
Time series foundation models face a scaling bottleneck because long-horizon forecasting is serial, while autoregressive rolling incurs computational overhead and error accumulation.
Method
Timer-S1 combines sparse TimeMoE and TimeSTP blocks for Serial-Token Prediction with trillion-point augmented pre-training and a multi-stage training pipeline.
Results
Timer-S1 achieves state-of-the-art forecasting performance on GIFT-Eval, with CRPS 0.485 and MASE 0.693.
Takeaways & Limitations
Timer-S1 provides a scalable time series foundation model that improves long-term forecasting while producing multi-step predictions without rolling-style autoregression.
Takeaways & Limitations
Timer-S1 does not natively incorporate exogenous covariates, leaving room for future improvement.
Abstract
from arXiv · showhide
We introduce Timer-S1, a strong Mixture-of-Experts (MoE) time series foundation model with 8.3B total parameters, 0.75B activated parameters for each token, and a context length of 11.5K. To overcome the scalability bottleneck in existing pre-trained time series foundation models, we perform Serial Scaling in three dimensions: model architecture, dataset, and training pipeline. Timer-S1 integrates sparse TimeMoE blocks and generic TimeSTP blocks for Serial-Token Prediction (STP), a generic training objective that adheres to the serial nature of forecasting. The proposed paradigm introduces serial computations to improve long-term predictions while avoiding costly rolling-style inference and pronounced error accumulation in the standard next-token prediction. Pursuing a high-quality and unbiased training dataset, we curate TimeBench, a corpus with one trillion time points, and apply meticulous data augmentation to mitigate predictive bias. We further pioneer a post-training stage, including continued pre-training and long-context extension, to enhance short-term and long-context performance. Evaluated on the large-scale GIFT-Eval leaderboard, Timer-S1 achieves state-of-the-art forecasting performance, attaining the best MASE and CRPS scores as a pre-trained model. Timer-S1 is released to facilitate further research.
1 Introduction
Timer-S1 addresses the scalability challenge of time series forecasting by combining serial forecasting with architecture, data, and post-training scaling. Its Serial-Token Prediction design adds serial computations without rolling-style autoregression, and the model achieves state-of-the-art GIFT-Eval performance.
- Time series forecasting is difficult because heterogeneous, multiscale, non-stationary, and stochastic signals complicate learning across domains.
- Timer-S1 uses an 8.3B-parameter sparse Mixture-of-Experts architecture with 0.75B activated parameters per token.
- Serial-Token Prediction uses TimeSTP blocks that reference the initial lookback series and intermediate representations to produce shift-by-one predictions progressively.
- Serial Scaling combines serial forecasting, data augmentation on the trillion-point TimeBench corpus, and post-training to enhance model capability.
- TimeSTP blocks remain active during inference, enabling multi-step forecasts in one forward pass without rolling-style autoregression.
- Timer-S1 achieves state-of-the-art GIFT-Eval performance with CRPS 0.485 and MASE 0.693.
2 Background
Time series foundation models seek broad generalization, but scaling remains difficult because forecasting is inherently serial. Timer-S1 responds with serial-token modeling, trillion-scale augmented data, and continued pre-training aimed at different forecasting capabilities.
- Time series forecasting has progressed from statistical and machine-learning methods to deep learning and foundation models trained on larger, diverse datasets.
- Foundation models address poor cold-start and data-scarce generalization by pre-training on varied evolving patterns before downstream adaptation.
- Scaling time series foundation models is bottlenecked because long-horizon accuracy depends on progressive serial forecasting.
- Autoregressive forecasting respects serial structure but incurs computational overhead and error accumulation through repeated rolling steps.
- Timer-S1 uses trillion-scale real and synthetic data with augmentation and continued pre-training to improve general evolving-pattern recognition and targeted capabilities.
3 Timer-S1: Architecture
Timer-S1 combines re-normalized patch tokens, sparse TimeMoE Transformer blocks, and TimeSTP blocks that add serial computation for multi-horizon forecasting. Its design supports quantile prediction while retaining serial forecasting during inference without rolling autoregression.
- 3.3 Forecasting Head: A shared forecasting head projects hidden embeddings into quantile predictions for future patches, with outputs de-normalized using the input mean and standard deviation.The head is trained with quantile loss related to the weighted quantile loss used for CRPS evaluation.
- 3.1 Normalization and Embedding: Timer-S1 re-normalizes each univariate series, divides it into fixed-length patches, and embeds the patches as Transformer tokens.Padding and a binary mask handle input lengths that are not divisible by the patch size.
- 3.2 Transformer Backbone: The model uses a decoder-only Transformer with TimeMoE blocks for heterogeneous time-series patterns and TimeSTP blocks for serial-token prediction.TimeMoE blocks contain multi-head attention and sparse expert modules, while TimeSTP adds serial computations after the main blocks.
- 3.2 Transformer Backbone: The sparse MoE backbone routes each token to selected experts, with E routed experts and K activated experts, while auxiliary loss balances expert usage.The reported configuration uses many total experts but activates only a few, supporting billion-scale parameters and faster inference.
- 3.2 Transformer Backbone: TimeSTP progressively reuses preceding-block embeddings while attending to initial input embeddings to predict successive future patch tokens.The architecture appends H TimeSTP blocks, each combining a projection layer with a TimeMoE block.
- 3.2 Transformer Backbone: TimeSTP blocks remain active during inference, enabling multiple patches in one forward pass and adaptive inference depth for the requested horizon.This avoids rolling-style autoregression and unnecessary computation from fixed-depth prediction.
4 Training
Timer-S1 combines large-scale data curation with serial-token pre-training and post-training to address time-series heterogeneity, predictive bias, and short- and long-horizon forecasting needs.
- Data Curation: TimeBench contains 1032 billion regularly sampled time points curated through preprocessing, predictability assessment, and multi-domain data collection.The corpus includes real-world and synthetic series, with quality checks such as causal imputation and outlier removal.
- Data Augmentation: Data augmentation targets predictive bias from imbalanced real-world distributions through frequency resampling and value-flipping.Resampling exposes the model to diverse temporal resolutions, while value-flipping reverses trends without changing temporal dependencies.
- Pre-Training: Timer-S1 uses a sparse Mixture-of-Experts architecture with 32 routed experts and 2 activated experts, alongside TimeSTP blocks for multi-patch prediction.The first-stage objective equally weights next-token and serial-token prediction and assigns equal weights across forecasting horizons.
- Post-Training: Post-training revisits GIFT-Eval Pretrain and TimeBench data, applies weighted Serial-Token Prediction to prioritize shallow blocks, and extends context from 2880 to 11520.The weighted objective emphasizes short-term forecasting, while mixed-data sampling is intended to mitigate overfitting to the post-training distribution.
- Training Infrastructure: The training pipeline is implemented with a hybrid memory-disk loading strategy that partitions TimeBench into 50MB shards.The shard size balances I/O concurrency and sampling randomness for the approximately 4TB stored corpus.
5 Experiments
Timer-S1 is evaluated across benchmark performance, scaling behavior, training objectives, architecture variants, augmentation, and pre-training. It achieves strong GIFT-Eval results, with advantages concentrated in medium- and long-term forecasting and supported by serial forecasting and multi-stage training.
- Benchmark Results: Timer-S1 performs substantially better on medium- and long-term forecasting tasks than on shorter-term settings.The analysis groups GIFT-Eval results by forecasting term length.
- Benchmark Results: Continued pre-training and long-context extension improve performance beyond a single pre-training stage.The post-training pipeline shifts focus toward short-term tasks through a weighted objective after fully training TimeSTP across horizons.
- Scaling Analysis: Serial-token prediction outperforms next-token and multi-token prediction under comparable block budgets while reducing rolling iterations and associated computation.Timer-S1 needs only a single TimeSTP block for the next prediction, whereas next-token prediction passes through the whole model.
- Scaling Analysis: Performance continues to improve up to 24 TimeMoE blocks and 16 TimeSTP blocks, reaching a billion-scale configuration.The study varies TimeMoE and TimeSTP depth separately during pre-training.
- Ablation Study: Removing serial forecasting or using shifted tokens worsens performance, while resampling augmentation improves robustness across temporal frequencies.A fixed patch size of approximately 16 produces an error spike on sinusoidal signals.
- Ablation Study: Pre-training on TimeBench substantially improves GIFT-Eval results compared with training from scratch under the same model configuration and post-training dataset.The result supports transfer of temporal patterns learned during pre-training.
6 Conclusion
Timer-S1 addresses time series foundation-model scalability with Serial Scaling, centered on Serial-Token Prediction and supported by large-scale pre-training, augmentation, and multi-stage training. It achieves state-of-the-art GIFT-Eval performance, while lacking native exogenous-covariate support and requiring broader adaptive representation learning.
- Conclusion: Timer-S1 is a billion-scale Mixture-of-Experts foundation model that addresses scalability bottlenecks through Serial Scaling.Its core innovation is Serial-Token Prediction, which respects the serial nature of forecasting and improves long-term forecasting performance.
- Conclusion: Pre-training on more than one trillion TimeBench points, data augmentation, and multi-stage training support the model’s forecasting approach.These components are presented as part of Timer-S1’s overall design.
- Conclusion: Timer-S1 achieves state-of-the-art performance on the GIFT-Eval leaderboard.The conclusion identifies leaderboard performance as the model’s principal outcome.
- Limitations: Timer-S1 does not natively incorporate exogenous covariates and needs improved generality across varying input contexts and output horizons.The authors identify multivariate-data training and adaptive representation learning as future directions.