Source-linked AI summary
A decoder-only foundation model for time-series forecasting
Abhimanyu Das, Weihao Kong, Rajat Sen, Yichen Zhou
TL;DR
Time-series forecasting lacks the vocabulary and grammar that helped motivate large language model foundation models, raising whether pretrained temporal patterns transfer to unseen datasets. TimesFM pretrains a patched decoder-only model on real-world and synthetic time-series data, achieving close to supervised state-of-the-art zero-shot accuracy across diverse datasets. The authors also report that it supports varying contexts, horizons, and temporal granularities, while cautioning that critical uses may require human oversight, testing, or fine-tuning.
Problem
The paper asks whether a pretrained time-series model can forecast previously unseen datasets without dataset-specific covariates or additional training.
Method
TimesFM is a decoder-only attention model with input patching, pretrained on a large corpus of real-world and synthetic time-series data.
Results
TimesFM achieves close to state-of-the-art zero-shot accuracy across diverse unseen datasets, outperforming llmtime by more than 25% in mean scaled MAE.
Takeaways & Limitations
A 200M-parameter model trained on O(100B) timepoints can provide practical zero-shot forecasting across different history lengths, prediction lengths, and granularities.
Takeaways & Limitations
For critical use cases, the authors recommend human-in-the-loop deployment or extensive testing and fine-tuning because some inputs may produce poor forecasts or hallucinations.
Abstract
from arXiv · showhide
Motivated by recent advances in large language models for Natural Language Processing (NLP), we design a time-series foundation model for forecasting whose out-of-the-box zero-shot performance on a variety of public datasets comes close to the accuracy of state-of-the-art supervised forecasting models for each individual dataset. Our model is based on pretraining a patched-decoder style attention model on a large time-series corpus, and can work well across different forecasting history lengths, prediction lengths and temporal granularities.
1 Introduction
Time-series forecasting is important across many domains, while progress in NLP motivates asking whether pretrained models can transfer temporal patterns to unseen forecasting datasets. TimesFM addresses this question with a large corpus and patched decoder architecture aimed at strong zero-shot forecasting.
- Time-series forecasting supports applications including supply chains, energy, traffic, weather, finance, manufacturing, healthcare, and natural sciences.
- Large language models demonstrate that massive pretraining can support diverse downstream tasks in zero-shot settings.
- The central question is whether large pretrained models can learn temporal patterns useful for forecasting on previously unseen datasets.
- TimesFM is a single foundation model that achieves close to state-of-the-art zero-shot accuracy across unseen datasets and varying histories, horizons, and granularities.
- 200M parameters and O(100B) timepoints are sufficient for a practical time-series foundation model whose zero-shot performance approaches fully supervised methods.
2 Related Work
Prior forecasting research largely uses supervised, dataset-specific training, while recent work reuses or fine-tunes language models. TimesFM targets the less-developed setting of one pretrained model with strong zero-shot performance across many datasets.
- Forecasting approaches include local univariate, global, and hybrid models, with deep learning typically applied when large training datasets are available.
- Earlier transfer-learning work studies dataset-to-dataset adaptation, but does not aim to train one foundation model for many datasets.
- Recent studies benchmark or fine-tune pretrained language models for forecasting, mostly on target datasets rather than through broad zero-shot pretraining.
- TimeGPT-1 is identified as the only other parallel zero-shot forecasting foundation model, but its access and several model and benchmark details are unavailable.
3 Problem Definition
The task is to build a general-purpose zero-shot forecaster that maps a past context of time points to a future horizon without dataset-specific covariates. Prediction accuracy is measured by closeness to actual values, such as MAE.
- The forecaster receives the past C time-points as context and predicts the future H time-points.
- Because the model is pretrained once for general use, it cannot use dataset-specific dynamic or static covariates during training.
- Mean Absolute Error (MAE) is one metric for measuring prediction closeness to actual values.
4 Model Architecture
TimesFM uses patched inputs, causal decoder-only transformers, masking, and residual output layers to support variable contexts and horizons. Longer output patches reduce autoregressive steps, while patch length introduces an efficiency–flexibility trade-off.
- Patching: Patching breaks a series into contiguous segments, improves performance, and reduces transformer tokens by the patch length factor.
- Decoder-only model: Decoder-only training predicts each next patch from all preceding patches and supports varying numbers of observed input patches.
- Longer output patches: Longer output patches provide a middle ground between full-horizon prediction and one-step autoregressive decoding when the horizon is unknown.
- Longer output patches: With input patches of 32 and output patches of 128, a 256-step forecast uses two generation steps instead of eight.
- Patch Masking: Randomly masking patch prefixes and complete initial patches exposes the model to all context lengths rather than only patch-length multiples.
- Input and transformer layers: Input patches become model-dimensional residual-block vectors with positional encodings, then pass through stacked multi-head causal-attention and feed-forward layers.
- Output layers and loss: Output tokens are mapped by a residual block to forecasts for the subsequent time window, and training minimizes point-forecasting MSE.
- Loss function: Probabilistic forecasting can use multiple quantile-loss heads or distribution-logit outputs with maximum-likelihood loss.
5 Pretraining Details
TimesFM is pretrained on a large, diverse corpus combining real-world and synthetic time series across domains and temporal granularities. Dataset mixing balances these sources and granularities during training.
- Data sources: The pretraining corpus combines Google Trends, Wiki Pageviews, synthetic series, and other public datasets across multiple domains and granularities.The corpus includes real-world data from search, Wikimedia, M4, Electricity, Traffic, and Weather, plus synthetic ARMA, seasonal, trend, and step-function series.
- Real-world data: Google Trends data covers roughly 22k head queries at hourly, daily, weekly, and monthly granularities.The selected query histories span 2007–2022, with hourly data from January 2018 to December 2019.
- Real-world data: Wiki Pageviews are cleaned, aggregated by page into four granularities, and filtered for excessive zeros; the final corpus contains roughly 300B time-points.
- Synthetic data: Synthetic data contributes 3M series of length 2048 generated from ARMA processes, seasonal patterns, trends, and step functions.A synthetic series may add one or more generated processes together.
- Dataset mixing and training: Training samples 80% real and 20% synthetic data, while weighting real data equally across hourly/sub-hourly, daily, weekly, and monthly groups.The model uses a maximum context length of 512 when the time series is long enough.
6 Empirical Results
TimesFM is evaluated zero-shot on held-out forecasting benchmarks and generally matches or exceeds strong supervised baselines. Ablations examine scaling, decoding, patch length, and dataset composition, with synthetic data helping underrepresented granularities.
- Evaluation setup: Held-out zero-shot evaluation tests one pretrained model across public datasets spanning domains, sizes, granularities, and horizon lengths.The benchmark groups are Darts, Monash, and Informer datasets, compared against baselines tuned or trained for specific tasks.
- Monash: TimesFM is the top Monash model, slightly better than N-BEATS within significance, and improves on llmtime’s performance by more than 25%.The comparison uses mean scaled MAE aggregated across the datasets.
- Darts: On Darts, TimesFM is within statistical significance of llmtime and seasonal ARIMA, while standard errors prevent a clear model ordering.The group contains only eight individual time series, and ARIMA required manually encoding seasonality for best results.
- Informer datasets: On eight ETT tasks with horizons 96 and 192 from context length 512, TimesFM performs best and PatchTST is within significance.Other long-horizon methods perform substantially worse despite training on these datasets.
- Ablation: A 32-point input patch gives strong performance while training almost twice as fast as a 16-point patch, making it the authors’ practical choice.Performance peaks around patch lengths 16 and 32 and increases toward neither endpoint across lengths 8–128.
- Ablation: Synthetic data improves performance on underrepresented granularities, especially the 15-minute ETTm datasets, while hourly ETTh results show almost no difference.Removing synthetic data also causes a performance drop on Monash.
- Downstream evaluation: A finetuning study reports that TimesFM performs better than all baselines on every reported dataset.
7 Conclusion
TimesFM is presented as a practical forecasting foundation model whose zero-shot performance comes close to fully supervised models across diverse time-series data.
- TimesFM achieves zero-shot forecasting performance close to fully supervised forecasting models on diverse time-series data.The model is pretrained on real-world and synthetic datasets comprising O(100B) timepoints.
8 Impact Statement
The paper discusses potential societal benefits and risks of a zero-shot forecasting foundation model, including privacy, bias, training cost, and reliability considerations.
- The model may enable downstream applications through zero-shot forecasting across a variety of tasks.The paper frames this potential as creating exciting possibilities while emphasizing the need to consider ethical and societal concerns.
- Data Privacy: Most data sources are publicly available and aggregated, and Google Trends data is differentially private.
- Bias: Biases in training data may persist in forecasts and lead to unfair real-world outcomes, although the absence of covariates reduces some sensitivities.
- The authors plan to release dataset details, model weights, and a model card to support analysis and finetuning with more diverse data.
- Training cost: The largest model has 200M parameters, while final training used 16 core TPUv5e for 2 days; experimentation and trial runs cost more.
- Inputs exist on which the model may perform poorly or hallucinate, motivating human-in-the-loop use, broad testing, or finetuning in critical applications.
A.1 Limitations and Future Work
The paper identifies limitations and future directions involving prompt tuning, probabilistic forecasting, covariates, finetuning, architecture choices, and interpretability.
- Prompt Tuning: Prompt-tuning techniques for improving inaccurate time-series foundation-model forecasts remain less clear than analogous techniques in LLMs.The current approach can tune simple hyperparameters such as context length.
- Probabilistic Forecasting: Probabilistic forecasting was not a main focus, although the framework could be trained with probabilistic loss functions and extended during finetuning.
- Covariate handling: The model is not pretrained with covariates because large volumes of data with meaningful covariates are difficult to obtain.The paper suggests residual regression at inference time and covariate use during finetuning as possible approaches.
- More finetuning studies: More extensive finetuning studies, including studies with covariates, are left for future work.
- Other architectures: The authors performed limited pretraining hyperparameter tuning and identify all-MLP and efficient linear state-space models as alternative architectures.
- Interpretability: Deep foundation models trained on large corpora may be less interpretable than statistical methods, and attribution methods do not fully solve this problem.
A.2 Metrics
The paper defines its reported metrics and explains how results are aggregated across datasets with different scales.
- The paper reports a defined set of metrics for evaluating forecasting results.
- For Monash benchmarks, ϵ = 0.1 avoids undefined values in normalized metrics such as MAPE.For multivariate datasets, metrics are computed per series and then averaged or median-aggregated; this paper uses means.
- Aggregating across datasets: Across datasets, each baseline metric is scaled by the naive-baseline metric because unnormalized MAE values have incompatible scales.The naive baseline repeats the last observed value across the prediction horizon.
A.3 Finetuning study on ETT
The experiments examine TimesFM’s zero-shot and limited-data forecasting behavior across ETT, Monash, and Darts datasets, alongside matched PatchTST and baseline comparisons. Results indicate strong performance on seasonal datasets, while PatchTST’s zero-shot behavior depends on pretraining context lengths and compute allocation.
- Finetuning study on ETT: 10% of the original training set is used for training or finetuning in the ETT comparison.The comparison reports MAE for different methods under the same limited-data setting.
- PatchTST ablation: PatchTST(ZS) performs poorly on Monash, where pretraining predominantly uses context lengths of 512 rather than the shorter contexts common in Monash.The authors also note that PatchTST(ZS) performs similarly to TimesFM(ZS) and PatchTST on ETT.
- PatchTST ablation: PatchTST’s zero-shot pretraining theoretically requires all possible context and horizon lengths, making maximum-performance pretraining more compute-intensive and tuning-sensitive than TimesFM.This comparison concerns the encoder-decoder PatchTST model used in the ablation.
- ETT: TimesFM and PatchTST are the best-performing models on the eight ETT tasks covering prediction horizons 96 and 192.Figure 4 aggregates MAE across four ETT datasets and two horizons.
- Darts: TimesFM performs well across the eight Darts datasets and is, on average, within the significance level of the best model.The Darts evaluation has wide confidence intervals because it contains only eight time-series overall.
- Illustrative examples: TimesFM captures trend and seasonal components in synthetic and real-world examples, including amplitude growth, seasonal peaks, and robustness to contextual outliers.In the illustrative examples, ARIMA and llmtime fail on some synthetic instances, while llmtime is thrown off by outliers in a traffic example.