Source-linked AI summary
Lag-Llama: Towards Foundation Models for Probabilistic Time Series Forecasting
Kashif Rasul, Arjun Ashok, Andrew Robert Williams, Hena Ghonia, Rishika Bhagwatkar, Arian Khorasani, Mohammad Javad Darvishi Bayazi, George Adamopoulos, Roland Riachi, Nadhir Hassen, Marin Biloš, Sahil Garg, Anderson Schneider, Nicolas Chapados, Alexandre Drouin, Valentina Zantedeschi, Yuriy Nevmyvaka, Irina Rish
TL;DR
Probabilistic time series forecasting lacks general-purpose foundation models despite their success in other modalities and the practical importance of uncertainty-aware forecasts. Lag-Llama uses a decoder-only transformer with lag covariates, pretrained on diverse time series, and shows strong zero-shot, few-shot, and fine-tuned performance on unseen datasets. The authors identify larger corpora, larger models, and multivariate extensions as directions for future work.
Problem
Most probabilistic time series forecasting methods are dataset-specific, while foundation models for time series remain underdeveloped despite their success in other modalities.
Method
Lag-Llama is a univariate probabilistic forecasting foundation model using a decoder-only transformer with lag features, trained on a large diverse corpus.
Results
Lag-Llama shows strong zero-shot generalization on unseen datasets, state-of-the-art performance after fine-tuning, and strong few-shot adaptation across varying data histories.
Takeaways & Limitations
Lag-Llama emerges as a general-purpose forecasting model that performs comparably to dataset-specific models without downstream-dataset knowledge.
Takeaways & Limitations
The work remains limited by the size of available open time series repositories and focuses on univariate rather than multivariate forecasting.
Abstract
from arXiv · showhide
Over the past years, foundation models have caused a paradigm shift in machine learning due to their unprecedented capabilities for zero-shot and few-shot generalization. However, despite the success of foundation models in modalities such as natural language processing and computer vision, the development of foundation models for time series forecasting has lagged behind. We present Lag-Llama, a general-purpose foundation model for univariate probabilistic time series forecasting based on a decoder-only transformer architecture that uses lags as covariates. Lag-Llama is pretrained on a large corpus of diverse time series data from several domains, and demonstrates strong zero-shot generalization capabilities compared to a wide range of forecasting models on downstream datasets across domains. Moreover, when fine-tuned on relatively small fractions of such previously unseen datasets, Lag-Llama achieves state-of-the-art performance, outperforming prior deep learning approaches, emerging as the best general-purpose model on average. Lag-Llama serves as a strong contender to the current state-of-art in time series forecasting and paves the way for future advancements in foundation models tailored to time series data.
1. Introduction
Probabilistic time series forecasting supports decision-making under uncertainty across many application domains, but most existing methods are dataset-specific. Lag-Llama addresses this gap as a pretrained foundation model evaluated on unseen datasets, with strong zero-shot, few-shot, and fine-tuned performance.
- Probabilistic forecasts provide decision-makers with multiple future scenarios and their likelihoods across applications including finance, weather, brain imaging, and computer systems.
- Lag-Llama is a foundation model for univariate probabilistic forecasting, pretrained on a broad corpus and evaluated on unseen time series datasets.
- Lag-Llama performs comparably or better than state-of-the-art dataset-specific models when downstream datasets provide different amounts of history.
- After fine-tuning, Lag-Llama achieves state-of-the-art performance across diverse domains and becomes the best general-purpose model without downstream-dataset knowledge.
- The work investigates zero-shot generalization, few-shot adaptation, pretraining-corpus diversity, and scaling with pretraining data.
2. Related Work
Related work spans statistical forecasting, neural architectures, and foundation models. The paper situates Lag-Llama within the shift toward transferable models pretrained on diverse data.
- Statistical forecasting includes ARIMA, ETS, and Theta models, while neural forecasting has expanded through recurrent, LSTM, and transformer architectures.
- Transformer-based forecasting methods differ in how they process time series inputs and transform outputs into forecasts.
- Foundation models use self-supervised or unsupervised learning on large datasets and have demonstrated adaptability across modalities and novel tasks.
- Scaling model size, dataset size, and data diversity has been associated with stronger transfer capabilities and few-shot learning on novel datasets.
3. Probabilistic Time Series Forecasting
The forecasting problem models the joint distribution of future values in a univariate sequence given its observed history and covariates. Training uses fixed context windows and an autoregressive factorization of the predictive distribution.
- A dataset contains D univariate time series sampled at discrete time points, with each series length denoted by T_i.
- The task is to model the joint distribution of P future values conditioned on a sequence’s observed past and covariates.
- Rather than using each series’ complete history, the model samples fixed context windows of size C to predict the next P values.
- The predictive distribution is approximated with a neural autoregressive model using the chain rule of probability.
4. Lag-Llama
Lag-Llama tokenizes univariate series with lagged values, temporal covariates, and summary statistics, then processes them with a decoder-only transformer to produce probabilistic forecasts. Robust standardization, autoregressive decoding, distributional outputs, and augmentation support training across heterogeneous time-series data.
- 4.1. Tokenization: Lag Features: Lag-Llama constructs tokens from lagged values spanning multiple frequencies and adds date-time features derived from each timestep.The lag indices can cover quarterly, monthly, weekly, daily, hourly, and second-level frequencies.
- 4.2. Architecture: The decoder-only transformer projects covariate-augmented tokens through masked layers and predicts forecast-distribution parameters for the next timestep.Inference uses greedy autoregressive decoding to generate simulated trajectories through the chosen prediction horizon.
- 4.3. Choice of Distribution Head: The distribution head maps model features to parametric distribution parameters, using a Student’s t-distribution with degrees of freedom, mean, and scale in the experiments.The authors retain this simple head while leaving more expressive alternatives for future work.
- 4.4. Value Scaling: Robust standardization removes the median and scales each context window by its interquartile range to reduce sensitivity to outliers.Mean and variance or related summary statistics are incorporated as time-independent covariates, while sampled values are de-standardized after prediction.
- 4.5. Training Strategies: Pretraining uses stratified corpus sampling plus Freq-Mix and Freq-Mask augmentation to reduce overfitting across heterogeneous datasets.The augmentation hyperparameters are included in the model’s hyperparameter search.
5. Experimental Setup
The experiments evaluate Lag-Llama on diverse unseen time-series datasets using zero-shot and fine-tuned settings, comparing it with statistical and deep-learning baselines. Performance is measured with CRPS during autoregressive probabilistic inference, while training uses validation-based model selection and supports variable prediction lengths.
- 5.1. Datasets: The study uses a corpus of 27 datasets spanning six grouped domains, with held-out datasets testing few-shot generalization and separate domains testing broader transfer.The grouped domain label does not imply a shared source or common data distribution.
- 5.2. Baselines: Lag-Llama is compared with statistical models and deep neural networks in a univariate forecasting setup.Other deep models are adapted for probabilistic forecasting with distribution heads and features comparable to Lag-Llama.
- 5.2. Baselines: Table 1 reports CRPS for zero-shot and fine-tuned Lag-Llama against supervised baselines trained separately on each unseen dataset, with lower values preferred.The table marks the best and second-best results for comparison.
- 5.3. Hyperparameter Search and Model Training Setups: Model selection searches 100 hyperparameter configurations and uses validation loss, with early stopping during both pretraining and dataset-specific fine-tuning.The decoder-only design allows downstream prediction lengths to vary.
- 5.3. Evaluation: Inference samples autoregressively from a context of length C until each dataset’s prediction horizon P, and CRPS averages performance across horizons and series.The evaluation uses 100 empirical samples.
6. Results
Lag-Llama performs comparably to supervised baselines without downstream data and improves substantially after fine-tuning, including state-of-the-art results on several unseen datasets. Its adaptation remains strong across different amounts of available history, though the most dissimilar domain requires more history.
- 6.1. Zero-Shot & Finetuning Performance on New Data: Fine-tuned Lag-Llama achieves the best average rank of 2.786, compared with 6.714 for zero-shot forecasting, and reaches state-of-the-art performance on three datasets.The fine-tuned model achieves state-of-the-art performance on ETT-M2, weather, and requests.
- 6.1. Zero-Shot & Finetuning Performance on New Data: Zero-shot Lag-Llama performs comparably to supervised baselines, with an average rank of 6.714 across unseen datasets.The comparison uses supervised baselines trained on specific datasets and pretrained Lag-Llama evaluated without downstream samples.
- 6.1. Zero-Shot & Finetuning Performance on New Data: On an entirely new domain and unseen frequency, Lag-Llama is comparable to baselines zero-shot and similar to state of the art after fine-tuning.This result is reported for the exchange-rate dataset.
- 6.1. Zero-Shot & Finetuning Performance on New Data: Qualitative forecasts closely match ground truth, and fine-tuning visibly improves forecast quality over the zero-shot setting.The comparison is shown in Figures 8 and 11.
- 6.2. Few-Shot Adaptation Performance on Unseen Data: Across 20%, 40%, 60%, and 80% history availability, Lag-Llama achieves the best average rank and increasingly better performance as more history becomes available.The evaluation restricts training data to the last K% of each dataset’s history, with K equal to 20, 40, 60, and 80.
- 6.2. Few-Shot Adaptation Performance on Unseen Data: When the downstream data is most dissimilar to pretraining, Lag-Llama requires increasing amounts of history to adapt and then performs comparably to state of the art.Lag-Llama is most often outperformed by TFT on the exchange-rate dataset.
7. Analysis
The analysis links broader and cross-domain pretraining coverage to greater diversity in time-series characteristics. It also examines how model validation loss scales with pretraining-related quantities.
- 7.1. Data Diversity: The pretraining corpus is analyzed with 22 catch22 features and PCA to quantify dataset diversity.The features were selected for classification ability from the hctsa library.
- 7.1. Data Diversity: Including multiple datasets within and across domains increases diversity in the top-two-component space of catch22 features.Diversity is assessed by averaging 22 canonical time-series characteristics per dataset and applying PCA.
- 7.2. Scaling Laws: The analysis fits neural scaling laws to Lag-Llama’s validation loss to study how performance scales with pretraining data and model parameters.The passage frames scaling-law analysis as a way to understand these quantities for Lag-Llama.
8. Discussion
Lag-Llama is presented as a general-purpose univariate probabilistic forecasting foundation model with strong zero-shot, fine-tuned, and few-shot adaptation performance. The discussion identifies larger corpora, larger models, and multivariate extensions as future directions.
- 8. Discussion: Lag-Llama achieves strong zero-shot generalization, state-of-the-art performance after fine-tuning, and strong few-shot adaptation across varying history amounts.It is reported as the best general-purpose model without downstream-data knowledge.
- 8. Discussion: Future work includes collecting larger open time-series corpora, scaling models beyond the explored sizes, and extending the approach from univariate to multivariate forecasting.The multivariate direction targets complex dynamics in real-world datasets.
9. Impact Statement
The paper notes potential benefits from better decision-making alongside possible negative societal impacts. It reports that the datasets are not known to contain or link to personally identifiable data.
- 9. Impact Statement: Potential societal consequences include optimizing processes through better decision-making as well as possible negative impacts.The statement does not specify the negative impacts.
- 9. Impact Statement: The authors report that, to their knowledge, none of the datasets contain or link to individual or personally identifiable data.The datasets were sourced from referenced locations.
10. Contributions
The project involved developing, evaluating, and releasing Lag-Llama through coordinated work on its architecture, datasets, experiments, and codebase.
- The team refined Lag-Llama’s architecture and training strategies, including lags, sampling, dropout, early stopping, and learning-rate scheduling.
- Large-scale pretraining, fine-tuning, and few-shot experiments were run and iterated across the project.
- The project integrated numerous models, datasets, experimental utilities, and open-source release support into the broader codebase.
- Contributors also developed experiment-specific integrations for OneFitsAll, N-BEATS, AutoGluon, and related forecasting baselines.
- Advisors and collaborators contributed feedback, writing, funding acquisition, and early research-direction development.
A. Details of Datasets
The experiments combine diverse time-series datasets spanning electrical demand, air quality, traffic, weather, finance, cloud systems, and other domains, with timestamp-based evaluation splits.
- The dataset collection spans diverse domains, including electricity, air quality, traffic, weather, exchange rates, transportation, and cloud computing.
- Individual datasets cover heterogeneous applications such as household electricity, transformer monitoring, sensor-based air quality, pedestrian counts, and serverless traces.
- The corpus includes varied temporal resolutions, from minute and hourly measurements to half-hourly, daily, and business-day series.
- Training and test splits are non-overlapping by timestamp, while pretraining validation uses the final 14 overlapping training windows from each dataset.
- The paper reports dataset domains, statistics, and hyperparameter choices in Tables 3–5, while supervised baselines use the same setup as Lag-Llama fine-tuning.
C.1. Results on the Pretraining Datasets
On datasets included in pretraining, Lag-Llama is not best on every individual dataset but achieves a comparable average rank and remains among the top-ranked models despite a shared training budget.
- The evaluation includes in-distribution datasets because a strong foundation model should perform both on pretrained data and on unseen distributions.
- Lag-Llama achieves a comparable average rank and is among the models with the top average ranks across pretraining datasets.
- Lag-Llama is not the best-performing model on every dataset because supervised baselines train exclusively on their respective datasets.
- The shared pretraining budget gives Lag-Llama less data per dataset than the dataset-specific supervised models.
- Forecast examples visualize median predictions and 50th- and 90th-percentile intervals for pretrained and downstream datasets, including zero-shot and fine-tuned settings.
- A neural scaling law fit to validation loss can extrapolate performance to larger pretraining-data regimes.