PaperlayerView exact revision on arXiv

Source-linked AI summary

Sundial: A Family of Highly Capable Time Series Foundation Models

Yong Liu, Guo Qin, Zhiyuan Shi, Zhi Chen, Caiyin Yang, Xiangdong Huang, Jianmin Wang, Mingsheng Long

arXiv:2502.00816v4cs.LG

TL;DR

Time series foundation models need flexible probabilistic forecasting because heterogeneous series make fixed parametric densities restrictive and many existing models lack generative prediction. Sundial uses TimeFlow Loss with minimally adapted Transformers and trillion-point TimeBench pre-training, achieving state-of-the-art zero-shot results on point and probabilistic benchmarks. Its generative forecasting provides multiple probable predictions and is intended to improve decision-making reliability.

  • Problem

    Existing time series foundation models are often non-probabilistic, while fixed prior densities can restrict distribution learning for heterogeneous series.

  • Method

    Sundial trains autoregressive Transformers with flow-matching-based TimeFlow Loss to predict next-patch distributions directly from continuous-valued, arbitrary-length series without discrete tokenization or specified prior densities.

  • Results

    Sundial achieves state-of-the-art zero-shot performance on point and probabilistic forecasting benchmarks including TSLib, GIFT-Eval, and FEV.

  • Takeaways & Limitations

    Sundial's generative forecasting capability supports multiple probable predictions and is intended to enhance reliability for real-world decision-making.

  • Takeaways & Limitations

    Pre-training uses a univariate paradigm with per-variable normalization and samples training contexts up to a maximum context length of 2880.

Abstract

from arXiv · show

We introduce Sundial, a family of native, flexible, and scalable time series foundation models. To predict the next-patch's distribution, we propose a TimeFlow Loss based on flow-matching, which facilitates native pre-training of Transformers on continuous-valued time series without discrete tokenization. Conditioned on arbitrary-length time series, our models are pre-trained without specifying any prior distribution and can generate multiple probable predictions, achieving more flexibility in representation learning than using parametric densities. Towards time series foundation models, we leverage minimal but crucial adaptations of Transformers and curate TimeBench with one trillion time points, comprising mostly real-world datasets and synthetic data. By mitigating mode collapse via TimeFlow Loss, we pre-train a family of Sundial models on TimeBench, which achieve unprecedented model capacity and generalization performance. In addition to excellent scalability, Sundial achieves state-of-the-art results on both point and probabilistic forecasting benchmarks with a just-in-time inference speed, i.e., making zero-shot predictions within a few milliseconds. We believe that Sundial's pioneering generative forecasting capability can improve model reliability in real-world decision-making. Code is available at: https://github.com/thuml/Sundial.

1. Introduction

Sundial addresses the limited probabilistic flexibility of time series foundation models with native generative forecasting on continuous-valued series. It combines TimeFlow Loss, Transformer adaptations, and trillion-scale pre-training to achieve state-of-the-art zero-shot forecasting results.

  • Time series forecasting is intrinsically non-deterministic, making varied probable predictions important for decision-making.
  • Most time series foundation models are not probabilistic forecasters, limiting their reliability in decision-making.
  • Sundial uses TimeFlow Loss and flow matching to learn each next-patch predictive distribution directly in the continuous-valued domain without prior parametric densities or discrete tokenization.
  • Sundial enhances Transformers with arbitrary-length patch tokenization, efficiency-oriented components, multi-patch prediction, and shared lookback representations for rapid multiple-sample generation.
  • 1 trillion time points: TimeBench supports pre-training and scaling-law validation for Sundial foundation models.
  • State-of-the-art zero-shot performance: Sundial performs strongly on point and probabilistic forecasting benchmarks including TSLib, GIFT-Eval, and FEV.

2. Related Work

Related work develops scalable time series foundation models and probabilistic forecasters, but parametric objectives can be too restrictive for heterogeneous data. Sundial extends generative modeling to autoregressive probabilistic forecasting with a loss designed for arbitrary distributions.

  • Time series foundation models address data-scarce scenarios through pre-training and support zero-shot forecasting with large model capacity.
  • Transformers dominate current large time series models, which adapt the architecture to time series dimensionality and heterogeneity.
  • Models such as TimesFM, Timer, and Time-MoE fit unimodal distributions with MSE or quantile loss, which can cause mode collapse and omit prediction confidence.
  • Generative modeling has been widely explored for foundation models but remains limited for time series foundation models beyond generation and task-specific forecasting.
  • TimeFlow Loss targets autoregressive conditional generation for arbitrary distributions, addressing the distinct forecasting structure of future time series.

3. Preliminaries

Flow-matching learns a continuous transformation from a simple source distribution to a target distribution, enabling generative forecasting without fixing a parametric prior. Sundial applies this approach through a generative network conditioned on learned historical representations to produce raw probabilistic forecasts.

  • 3.1. Flow-Matching: Flow-matching transforms samples from a source distribution into samples from a target distribution along a continuous probability path.The path is defined by a time-dependent velocity field and its associated ordinary differential equation.
  • 3.1. Flow-Matching: The flow-matching objective trains a network to regress the target velocity field, with conditional flow matching providing an equivalent optimization objective.
  • 3.1. Flow-Matching: A conditional optimal-transport path with a Gaussian source enables training on target samples and generation through a push-forward process.The interpolation uses t ∼ U[0, 1], x0 ∼ N(0, 1), and xt = tx1 + (1 − t)ϵ.
  • 3.2. Generative Models for Probabilistic Forecasting: Probabilistic forecasting predicts future series conditioned on a learned representation of historical observations, while a single parametric density can limit foundation-model scalability.The forecasting target is p(x_t+1:t+f|h_t), where h_t is learned from the lookback series.
  • 3.2. Generative Models for Probabilistic Forecasting: Sundial uses a small trainable generative network conditioned on learned representations to model the target distribution and sample raw predictions for probabilistic forecasting.The generated samples can also be used to calculate forecast statistics, with uncertainty modeled beyond prediction intervals.
  • 3.2. Generative Models for Probabilistic Forecasting: Sundial combines continuous patch embeddings, a decoder-only Transformer, and TimeFlow Loss to generate multiple plausible predictions under flow matching.

4. Approach

Sundial combines flexible time-series tokenization, an adapted Transformer backbone, and TimeFlow-based probabilistic generation. Its approach supports varied-length inputs, multi-patch prediction, and efficient repeated sampling conditioned on curated TimeBench data.

  • 4.1. Approach: Sundial comprises re-normalization and patch embedding, a Transformer backbone, and TimeFlow Loss for distribution modeling and raw-series generation.The model autoregressively learns token representations, while TimeFlow transforms random noise into non-deterministic predictions.
  • 4.1.1. Time Series Tokenization: Patch embedding pads non-divisible inputs, masks padded positions, and preserves continuous values without discrete quantization.A shared MLP embeds patches, reducing the Transformer's token context length.
  • 4.1.2. Transformer Backbone: The decoder-only Transformer uses Pre-LN, causal RoPE attention, FlashAttention, and KV Cache to produce representations from previous patch tokens.The attention projections map token embeddings into query, key, and value representations, while RoPE adds patch-position information.
  • 4.1.2. Transformer Backbone: Multi-patch prediction uses forecast length F > P during pre-training, reducing autoregression steps while accommodating different data frequencies.The model generates length-F predictions at each position from the Transformer's learned representations.
  • 4.1.3. TimeFlow Loss: TimeFlow Loss trains a flow-matching network conditioned on time-invariant representations, transforming initial Gaussian noise into predicted samples.The conditional path is constructed by conditional optimal transport, and the noised target is jointly processed with t.
  • 4.1.3. TimeFlow Loss: Inference advances Gaussian noise along a K-step uniform trajectory and reuses lookback representations across noises to estimate medians and quantiles efficiently.Repeated sampling produces probabilistic forecasts without repeated computation of the shared lookback representation.
  • 4.2. TimeBench: TimeBench contains over a trillion time points from varied sources, mostly real-world records, with 0.05% synthetic data for pattern diversity.It also includes meteorological data and multiple frequencies covering diverse temporal dynamics.

5. Experiments

Sundial is evaluated across zero-shot point and probabilistic forecasting benchmarks, scalability studies, generative quality analyses, adaptation experiments, and architectural ablations. It generally achieves strong accuracy and probabilistic performance while retaining fast inference and improving with model scale and selected Transformer adaptations.

  • Probabilistic forecasting: Sundial ranks first in MASE and second in CRPS among supervised and advanced foundation models across all unseen GIFT-Eval datasets.GIFT-Eval contains 23 datasets and evaluates zero-shot performance across diverse forecasting configurations.
  • Probabilistic forecasting: On FEV, Sundial exceeds 70% of statistical and supervised deep baselines, ranks second among zero-shot pre-trained models, and achieves a 35× inference speedup over Chronos.FEV includes 27 datasets not seen during pre-training; patch-wise tokenization and multi-patch prediction support the speed advantage.
  • Generative forecasting: Sampling from Sundial's predictive distribution supports arbitrary statistics beyond means and quantiles, while generated patterns remain eventful and coherent with the input series.The experiments also examine calibration as sample count and generation-step choices vary.
  • Scalability: The large Sundial model reduces converged training loss by 15.38% versus the small model, indicating improved utilization of increased capacity.Table 1 also reports consistently better downstream performance as parameters scale.
  • Model adaptation: Fine-tuning improves performance on unseen FEV test splits, while training from scratch performs worse, indicating knowledge transfer from pre-training.The model is tuned once on aggregated FEV datasets with varying short-term prediction lengths.

6. Conclusion

The conclusion presents TimeBench and Sundial as resources and methods for scaling time series foundation models. It emphasizes continuous tokenization and generative modeling as the paper's main advances, with strong forecasting leaderboard performance and released models supporting future work.

  • Dataset: TimeBench is a trillion-scale pre-training dataset curated from publicly available resources for developing time series foundation models.The conclusion frames the dataset as a resource for the research community.
  • Contributions: The paper argues that continuous patch tokenization is effective and efficient for time series, while generative modeling natively learns continuous-valued data distributions.These are presented as the two main contribution areas.
  • Contributions: TimeFlow Loss equips autoregressive models to sample from non-categorical distributions and accommodate heterogeneous time series distributions.The objective is positioned as a training approach for generative time series foundation models.
  • Results: Pre-trained Sundial models achieve substantial advances on recognized forecasting leaderboards and demonstrate notable zero-shot forecasting performance.The released models are intended to support broader applicability of generative forecasting.

A. Dataset Statistics

TimeBench addresses the limited scale and heterogeneity of existing time series pre-training corpora by combining extensive real-world data with preprocessing and statistical analysis. Its sources span multiple domains and exceed one trillion time points.

  • Motivation: Existing time series foundation-model pre-training remains relatively limited, motivating investigation of scaling under heterogeneous time series distributions.The passage contrasts time series heterogeneity with other modalities.
  • Curation: TimeBench construction required missing-value imputation, abnormality exclusion, normalization, and statistical analysis of intrinsic series properties.These preprocessing steps address the scarcity and uneven quality of openly available time series.
  • Composition: TimeBench exceeds one trillion time points by combining research datasets with real-world series from finance, IoT, meteorology, and healthcare.Most datasets are real-world records, while 0.05% is synthetic data used to increase pattern diversity.
  • Statistics: The dataset's key statistics are summarized in Table 4.The supplied passage identifies the table as the summary of TimeBench statistics.

B. Implementation Details

The implementation uses balanced multi-domain pre-training, task-dependent prediction lengths, and Transformer configurations tailored to scalable continuous time series modeling. The discussion also links TimeFlow to more diverse and coherent predictive distributions than alternative objectives.

  • Implementation: Training uses PyTorch, AdamW, 32 NVIDIA A100 GPUs, S3-format univariate pre-training, domain-ratio sampling, and global shuffling.The setup balances domain weights and diversity during training.
  • Training configuration: Sundial uses prediction length F = 16 for FEV and F = 720 for point forecasting and GIFT-Eval, with shorter requested horizons handled from the longer model output.These settings match the differing benchmark horizon ranges.
  • Mode collapse: Mode collapse limits output diversity when similar lookbacks lead to divergent trends, and MSE training can produce over-smooth predictions through a unimodal distribution assumption.This is identified as a central representation-learning challenge for heterogeneous time series.
  • Model configurations: The Sundial family configuration comparison reports architecture, model size, pre-training scale, token level, tokenization, context length, and probabilistic capability.These dimensions organize the model-family specifications.
  • Generative objective: TimeFlow models more coherent and diverse predictive distributions than alternative training objectives, evaluated using CRPS.The comparison targets distributional quality rather than only single-point accuracy.

C.2. Scaling Behavior Using More Data

Sundial’s scaling behavior is evaluated by training-data size and by varying inference lookback lengths, using averaged zero-shot results across four forecasting horizons on TSLib.

  • Scaling with more data: The comparison includes Sundial trained on subsets matching the 94-billion-point Chronos and 230-billion-point Moirai pre-training datasets.These subsets are used to highlight Sundial’s behavior as the training dataset becomes larger.
  • Scaling with more data: Sundial is compared across models trained on different dataset scales, with results averaged over horizons {96, 192, 336, 720} on Time-Series-Library.The comparison measures training data in time points, with 1B denoting one billion points.
  • Varying lookback lengths: Sundial’s inference context can be varied across lookback lengths from 480 to 2880, rather than being fixed during deployment.Figure 10 averages results across prediction lengths {96, 192, 336, 720} on TSLib.
  • Varying lookback lengths: Lookback-window size can be tuned to the forecasting horizon and data periodicity, although high-frequency data still requires stronger long-context capabilities.The authors characterize this adjustment as training-free during inference.

C.4. Zero-Shot Results of Point Forecasting

The paper evaluates Sundial’s zero-shot point forecasting against leading time series foundation models on held-out datasets spanning major forecasting benchmarks and horizons.

  • Time-Series-Library: Sundial is evaluated on Time-Series-Library horizons {96, 192, 336, 720}, using multiple model sizes and a fixed context length of 2880.Predictions are truncated for tasks requiring fewer than F = 720 steps.
  • Compared foundation models: The comparison uses official checkpoints of Time-MoE, Timer, Moirai, TimesFM, and Chronos on datasets excluded from the corresponding models’ pre-training.Each model uses its maximum input length during inference, and evaluation reports MSE and MAE.
  • GIFT-Eval: GIFT-Eval covers 23 datasets, 144,000 time series, 177 million data points, and 97 forecasting configurations for broad zero-shot evaluation.The paper uses the benchmark’s official evaluation suite and reports aggregated results.

D.1. Showcases of Sundial

The showcases compare Sundial’s generative zero-shot forecasts with a deterministic Transformer trained using the same backbone and TimeBench.

  • Zero-shot examples: Figures 11–13 show Sundial zero-shot forecasts across FEV and TSLib datasets, including leaderboard and long-term forecasting examples.The figures present examples from Sundial Base.
  • Zero-shot examples: Twenty predictions generated from different initial noise values are used to estimate the median and 80% prediction interval.This procedure visualizes forecast uncertainty rather than only a single trajectory.
  • Generative versus deterministic forecasting: Sundial can generate multiple future possibilities from a lookback series, whereas the MSE-trained Transformer outputs only a mean prediction.The comparison holds the backbone and TimeBench pre-training data constant while changing the forecasting objective.

E. Limitations

The paper identifies limitations involving high-frequency data, sampling strategy, multivariate information, and potential over-smoothing from autoregressive prediction.

  • Data-frequency scope: Performance on very high-frequency data is not guaranteed because TimeBench contains many middle- and low-frequency series.The paper proposes extending Sundial to multi-scale time series.
  • Reliability: Sundial may still face hallucinations despite progress in enlarging model capacity.This is stated as an unresolved limitation of the model family.
  • Sampling strategy: The current sampler begins from random Gaussian noise and leaves room for improved sampling and post-processing, including frequency normalization.The limitation concerns the sampling strategy rather than the forecasting objective itself.
  • Model adaptation: Univariate pre-training prevents Sundial from explicitly using variate correlations or covariate information.The paper identifies multivariate pre-training as a future direction for domain-specific models.
  • Model adaptation: Multiple autoregressive steps may still produce over-smoothed predictions and unreliable results.This remains a limitation despite autoregressive models’ flexible input context length.
Loading 2502.00816v4…