PaperlayerView exact revision on arXiv

Source-linked AI summary

Chronos: Learning the Language of Time Series

Abdul Fatir Ansari, Lorenzo Stella, Caner Turkmen, Xiyuan Zhang, Pedro Mercado, Huibin Shen, Oleksandr Shchur, Syama Sundar Rangapuram, Sebastian Pineda Arango, Shubham Kapoor, Jasper Zschiegner, Danielle C. Maddix, Hao Wang, Michael W. Mahoney, Kari Torkkola, Andrew Gordon Wilson, Michael Bohlke-Schneider, Yuyang Wang

arXiv:2403.07815v3cs.LGcs.AI

TL;DR

Chronos targets general-purpose forecasting beyond the conventional train-and-predict-on-one-dataset regime. It tokenizes time series and trains existing language-model architectures with cross-entropy, achieving strong in-domain performance and comparable or sometimes superior zero-shot performance on unseen datasets.

  • Problem

    Existing deep forecasters generally train and predict on the same dataset, while general-purpose forecasting is constrained by scarce, unevenly available time-series data.

  • Method

    Chronos scales and quantizes time series into a fixed vocabulary, trains existing transformer language-model architectures with cross-entropy, and supplements public data with synthetic and mixup augmentations.

  • Results

    Across 42 datasets, Chronos significantly outperforms existing local and task-specific deep-learning baselines in-domain and achieves comparable or occasionally superior zero-shot performance on new datasets.

  • Takeaways & Limitations

    Chronos can simplify forecasting pipelines by providing an inference-only alternative to training and tuning models for individual tasks.

  • Takeaways & Limitations

    Chronos can struggle with exponential trends and underestimate trends when the available context is short.

Abstract

from arXiv · show

We introduce Chronos, a simple yet effective framework for pretrained probabilistic time series models. Chronos tokenizes time series values using scaling and quantization into a fixed vocabulary and trains existing transformer-based language model architectures on these tokenized time series via the cross-entropy loss. We pretrained Chronos models based on the T5 family (ranging from 20M to 710M parameters) on a large collection of publicly available datasets, complemented by a synthetic dataset that we generated via Gaussian processes to improve generalization. In a comprehensive benchmark consisting of 42 datasets, and comprising both classical local models and deep learning methods, we show that Chronos models: (a) significantly outperform other methods on datasets that were part of the training corpus; and (b) have comparable and occasionally superior zero-shot performance on new datasets, relative to methods that were trained specifically on them. Our results demonstrate that Chronos models can leverage time series data from diverse domains to improve zero-shot accuracy on unseen forecasting tasks, positioning pretrained models as a viable tool to greatly simplify forecasting pipelines.

1 Introduction

Chronos addresses the lack of a unified general-purpose forecasting model by minimally adapting language-model architectures to time series. It combines tokenization, broad data augmentation, and pretraining, achieving strong in-domain and zero-shot results.

  • Forecasting supports decision-making across retail, energy, finance, healthcare, and climate science, yet deep forecasters typically train and predict on the same dataset.
  • Chronos reframes forecasting as language modeling by scaling and quantizing time-series values into discrete tokens.
  • Figure 1 depicts tokenization, language-model training with cross-entropy, and autoregressive sampling of multiple trajectories for predictive distributions.
  • Training combines public time-series datasets with TSMixup and Gaussian-process synthesis to address limited data quantity and quality.
  • Across 42 datasets, Chronos surpasses traditional models and task-specific deep-learning approaches for in-domain and zero-shot forecasting.
  • Chronos delivers zero-shot forecasts without task-specific adjustments, while its relatively modest size offers an alternative to larger computationally demanding models.

2 Background and Related Work

Time-series forecasting includes classical local models, global deep-learning models, and emerging LLM-based approaches. Chronos differs from prior work by targeting zero-shot probabilistic forecasting with minimal time-series-specific machinery.

  • Forecasting predicts the joint distribution of future values from a historical context, with Chronos focusing on univariate scalar observations.
  • Classical methods fit separate local models per series, whereas deep-learning global models learn across series within a dataset.
  • Language models predict the next token from a finite vocabulary, commonly using transformer encoder-decoder or decoder-only architectures.
  • Prior LLM forecasters use text prompts, specialized tokenization, patch embeddings, or task-specific fine-tuning, often requiring dataset-specific adaptation.
  • Zero-shot forecasting evaluates models on time series from unseen datasets; Chronos combines real and Gaussian-process synthetic data and produces probabilistic forecasts.
  • Other pretrained forecasting models use real-valued inputs and time-series-specific designs such as time features, lags, patching, and distribution heads.

3 Chronos: A Language Modeling Framework for Time Series

Chronos converts real-valued time series into tokens through scaling and quantization, then applies standard language-model training to forecast token sequences. Autoregressive sampling and dequantization produce probabilistic numerical forecasts.

  • Time Series Tokenization: Chronos maps real-valued observations to a finite vocabulary by first scaling each series and then quantizing values into bins.
  • Time Series Tokenization: Uniform binning is used because unseen datasets may have value distributions unlike the training data, but it restricts predictions to [c1, cB].
  • Model Architecture: Chronos can process token sequences with encoder-decoder or decoder-only language models without architectural changes beyond adjusting vocabulary size.
  • Objective Function: The model is trained with categorical cross-entropy on the predicted distribution of each next token, including EOS tokens.
  • Objective Function: Categorical modeling preserves arbitrary output distributions, including multimodal ones, while treating regression as classification over quantized bins.
  • Probabilistic Forecasting: At inference, Chronos autoregressively samples token paths, dequantizes them, and reverses scaling to obtain multiple numerical forecast trajectories.

4 Data Augmentation

The paper addresses limited and low-diversity public time-series data with TSMixup and KernelSynth, combining real series patterns and generating synthetic series through Gaussian processes.

  • Public time-series data are less abundant and diverse than NLP text datasets, challenging zero-shot forecasting models.
  • 4.1 TSMixup: Time Series Mixup: TSMixup extends Mixup by sampling and scaling multiple time series, then taking their convex combination to diversify training patterns.The method samples k series of a selected length from training datasets and mixes them after scaling.
  • 4.1 TSMixup: Time Series Mixup: TSMixup combines patterns from different time series, producing augmentations that improve pattern diversity.
  • 4.2 KernelSynth: Synthetic Data Generation using Gaussian Processes: KernelSynth supplements limited real data by generating synthetic time series with Gaussian processes.It is motivated by compositional Gaussian-process kernels that can represent diverse time-series patterns.
  • 4.2 KernelSynth: Synthetic Data Generation using Gaussian Processes: KernelSynth samples basis kernels, combines them with random addition or multiplication, and draws synthetic series from the resulting Gaussian-process prior.The resulting compositions can produce intricate patterns from simple basis kernels.

5 Experiments

Across 42 datasets, Chronos models outperform established baselines in-domain and deliver competitive zero-shot forecasts, while fine-tuning further improves performance on unseen datasets.

  • Main Results: Chronos models surpass classical statistical, task-specific deep learning, and other pretrained models across the 42-dataset evaluation.The evaluation includes 15 in-domain and 27 zero-shot datasets.
  • Benchmark I: In-domain Results: Larger Chronos-T5 models achieve the best aggregated scores and average ranks on the 15-dataset in-domain benchmark.They outperform local models, task-specific models such as PatchTST and DeepAR, and pretrained models such as Lag-Llama and Moirai-1.0-R.
  • Benchmark II: Zero-shot Results: On 27 unseen datasets, Chronos models outperform standalone local models and other pretrained models while matching the best task-specific deep learning models.Chronos-T5 (Large) ranks second for point forecasting and Chronos models place second through fourth for probabilistic forecasting.
  • Benchmark II: Zero-shot Results: Fine-tuning Chronos-T5 (Small) on individual Benchmark II datasets makes it the best-performing model on average.The fine-tuned model overtakes larger zero-shot Chronos models and the best task-specific models.
  • Analysis of Hyperparameters: Performance improves with model size across both in-domain and zero-shot benchmarks, but models larger than 710M parameters were not explored because inference would be impractical.The tested models range from 20M to 710M parameters.
  • Analysis of Hyperparameters: Synthetic data improves both in-domain and zero-shot metrics most consistently near a 10% training proportion, whereas larger proportions tend to worsen performance.The authors attribute this decline to synthetic Gaussian-process data not representing all real-world time series.

6 Discussion

Chronos shows strong zero-shot performance while simplifying deployment across heterogeneous forecasting settings, but its current scope and practical limitations remain important. The discussion highlights inference speed, limited public data, and extensions beyond univariate forecasting.

  • Chronos achieves remarkable zero-shot performance on a comprehensive collection of test datasets.
  • Beyond Zero-shot Univariate Forecasting: The study focuses on univariate, uniformly spaced forecasting, leaving exogenous information, irregular sampling, and multivariate forecasting outside its main scope.These settings are identified as common practical extensions requiring additional modeling.
  • Beyond Zero-shot Univariate Forecasting: Chronos representations may support classification, clustering, and anomaly detection, but these downstream applications remain future work.
  • Inference: Chronos can be deployed across diverse history lengths, frequencies, prediction horizons, and context lengths without task-specific training.This substantially simplifies forecasting pipelines compared with models trained separately for each task.
  • Inference: Larger Chronos models may have slower inference than task-specific deep learning models, although they are not prohibitively slow.Their inference speed is comparable to some statistical local models.
  • Data: High-quality public time-series data remains limited, creating a trade-off between training on more datasets and reserving datasets for zero-shot evaluation.The authors also report that Gaussian-process synthetic data improves model performance when included in training.

7 Conclusion

Chronos adapts existing language-model architectures for forecasting with minimal time-series-specific modification. It performs strongly both on training-domain datasets and on unseen datasets, supporting simpler forecasting pipelines.

  • Chronos adapts existing language-model architectures for time-series forecasting using scaling-and-quantization tokenization rather than time-series-specific architectures.
  • Chronos significantly outperforms existing local models and task-specific deep-learning baselines on in-domain forecasting.
  • Chronos performs competitively with the best deep-learning baselines trained on unseen datasets in zero-shot forecasting.The authors also report promising evidence that fine-tuning can improve results further.
  • Strong Chronos performance suggests pretrained language models can simplify forecasting pipelines without sacrificing accuracy.The proposed practical alternative is inference-only deployment instead of training and tuning a model for each task.

A Algorithms

The algorithms appendix identifies the procedures and kernel bank used for synthetic time-series generation in KernelSynth.

  • Algorithm 1 and Algorithm 2 provide pseudocode for TSMixup and KernelSynth, respectively.
  • Table 2 lists the kernel bank K used by KernelSynth.

B Datasets

The empirical study uses 55 datasets spanning multiple domains and frequencies, with separate roles for pretraining, in-domain evaluation, and zero-shot evaluation. The listed datasets cover energy, finance, macroeconomics, and other forecasting settings.

  • 55 datasets were used for experiments: 13 for pretraining only, 15 for in-domain evaluation, and 27 for zero-shot evaluation.
  • Energy: Spanish Energy and Weather combines electricity consumption, generation, pricing, and weather data across Spain.
  • Finance and economics: The financial and economic datasets include CIF 2016 banking data, exchange rates for eight countries, monthly FRED-MD indicators, and ATM cash withdrawals.
  • Finance and economics: Several datasets combine real and artificial series, including CIF 2016, where 24 series are real and 48 are artificially generated.

B.3 Healthcare

The healthcare datasets cover daily COVID-19 deaths and monthly patient counts related to medical products. They are included among the datasets partitioned for Chronos training and evaluation.

  • Covid Deaths contains daily COVID-19 death counts from countries and states between January and August 2020.
  • Hospital contains monthly patient-count time series for medical products from January 2000 through December 2006.
  • The dataset table partitions data into pretraining-only, in-domain evaluation, and zero-shot evaluation categories.

B.4 Nature

The nature-related collection spans weather, climate, environmental, transportation, tourism, economic, retail, and online activity series across multiple frequencies and geographies.

  • Weather and climate: Weather and climate datasets include temperature, rainfall, air quality, weather variables, climate indicators, and gridded WeatherBench measurements.These data range from station-level observations to 32×64 spatial grids and include hourly, daily, and weekly frequencies.
  • Other domains: Other datasets represent car-part sales, retailer SKU profits, pedestrian counts, and daily Wikipedia page views.These series span monthly, weekly, daily, and long-term observational settings.
  • Transportation: Transportation datasets cover bike stations, rideshare services, taxi rides, road occupancy, and Uber pick-ups.They include hourly, 30-minute, daily, and aggregated observations from Mexico City, New York, and the San Francisco Bay area.
  • Forecasting benchmarks: The collection includes tourism and forecasting-competition datasets spanning monthly, yearly, hourly, and other sampling periods.M1, M3, and M4 cover economic, industrial, demographic, financial, and other domains; M5 contains product sales through validation.

C Baselines

The benchmark compares Chronos with local statistical, task-specific deep-learning, and pretrained-model baselines under documented compute and configuration choices.

  • Baseline families: 17 baseline methods include local statistical models and task-specific deep-learning architectures such as DeepAR, PatchTST, TFT, DLinear, and WaveNet.The statistical implementations rely on StatsForecast, while the listed neural architectures provide task-specific comparisons.
  • Compute: Inference uses CPUs for local models and selected architectures, one V100 GPU for several deep-learning and pretrained models, and eight V100 GPUs for LLMTime.WaveNet and GPT4TS training used one NVIDIA V100 GPU, while other baselines used Intel-based CPU instances.
  • Configuration: Most baseline configurations were not dataset- or globally tuned, and stochastic task-specific models were trained and evaluated three times before averaging.The reported setup also specifies batch size, time limits, and early stopping for selected implementations.
  • Configuration: Statistical baselines use default StatsForecast hyperparameters with season lengths inferred from data frequency.Examples include season lengths of 7 for daily data and 24 for hourly data.
  • Model-specific settings: GPT4TS context length is set by a frequency-specific multiplier times the prediction length, while LLMTime uses a 512 context length.The GPT4TS rule is summarized in Table 4; LLMTime was evaluated only with Llama-2 70B under modified settings.
  • Documentation: Table 5 summarizes the baseline models and their hyperparameter choices, with unspecified values left at implementation defaults.

D Evaluation Metrics

Evaluation uses point and probabilistic forecasts over context and prediction horizons, measuring point accuracy with MASE and probabilistic quality with WQL.

  • Evaluation setup: The evaluation dataset contains N time series, each spanning context length C and prediction horizon H.Accuracy is evaluated on predictions from x_i,C+1:C+H across all series, using either point or probabilistic forecasts.
  • Point forecasts: MASE divides each series’ mean absolute error by the empirical error of a seasonal naïve model.The metric is scale-independent because its denominator scales proportionally to the series values, and dataset MASE averages across series.
  • Benchmark documentation: Table 5 documents baseline hyperparameter choices relevant to interpreting the benchmark configuration.
  • Probabilistic forecasts: WQL aggregates quantile loss over predicted quantiles and multiple series and prediction instants.The evaluation uses quantile levels from 0.1 through 0.9, so K = 9.

E Additional Results

Additional results document Chronos training costs and forecasting performance across in-domain and unseen datasets, with figures illustrating rankings, zero-shot synthetic-data evaluation, and forecast behavior. An inference correction substantially improved the reported Chronos results.

  • Benchmark results: Tables 7–10 report WQL and MASE scores for 15 in-domain and 27 unseen datasets, alongside aggregate relative scores and average ranks.Chronos and task-specific models were averaged over three random seeds.
  • Benchmark results: Figures 18 and 19 compare average model ranks on the in-domain and unseen benchmarks, respectively.Benchmark I contains 15 datasets included in Chronos training, whereas Benchmark II contains 27 datasets not seen during training.
  • Synthetic-data evaluation: Chronos-T5-Synth (Small) is evaluated zero-shot on both benchmarks because it was trained only on synthetic data.The comparison includes local and task-specific models.
  • Forecast examples: Chronos-T5 (Base) produces plausible forecasts and prediction intervals for AR(2) and AR(3) processes, while AR models obtain lower MSE on the simpler AR(2) process.The figure caption also describes comparisons with the ground-truth AR model, a correctly ordered fitted AR model, and AutoARIMA.
  • Corrections and updates: An off-by-one error in decoded bin indices had artificially worsened earlier Chronos results; fixing inference significantly improved them without retraining the models.The correction also changed the conclusion of the vocabulary-size experiment.
Loading 2403.07815v3…