PaperlayerView exact revision on arXiv

Source-linked AI summary

Large Language Models Are Zero-Shot Time Series Forecasters

Nate Gruver, Marc Finzi, Shikai Qiu, Andrew Gordon Wilson

arXiv:2310.07820v3cs.LG

TL;DR

The paper asks whether pretrained LLMs can forecast heterogeneous, uncertain time series without task-specific training. It encodes numerical values as text and adapts token probabilities into continuous densities, finding that LLMs can achieve competitive or superior zero-shot forecasting while supporting flexible uncertainty and auxiliary capabilities. However, performance depends on numerical tokenization and alignment, with GPT-4 showing worse forecasting and calibration than GPT-3.

  • Problem

    Time-series forecasting involves heterogeneous data, missing values, and futures that are highly uncertain, while prior LLM approaches commonly rely on fine-tuning.

  • Method

    LLMTIME encodes numerical time series as digit strings, uses pretrained LLMs for next-token prediction, and converts discrete token distributions into continuous densities.

  • Results

    LLMTIME matches or exceeds purpose-built time-series models across varied benchmarks in zero-shot evaluation, with strong likelihood, CRPS, and MAE performance.

  • Takeaways & Limitations

    LLM biases toward simple and repetitive patterns, flexible multimodal distributions, and language-based interaction can support forecasting and related explanatory functionality.

  • Takeaways & Limitations

    GPT-4 performs worse than GPT-3 for forecasting because altered tokenization and API restrictions hinder digit-level modeling and likelihood evaluation.

Abstract

from arXiv · show

By encoding time series as a string of numerical digits, we can frame time series forecasting as next-token prediction in text. Developing this approach, we find that large language models (LLMs) such as GPT-3 and LLaMA-2 can surprisingly zero-shot extrapolate time series at a level comparable to or exceeding the performance of purpose-built time series models trained on the downstream tasks. To facilitate this performance, we propose procedures for effectively tokenizing time series data and converting discrete distributions over tokens into highly flexible densities over continuous values. We argue the success of LLMs for time series stems from their ability to naturally represent multimodal distributions, in conjunction with biases for simplicity, and repetition, which align with the salient features in many time series, such as repeated seasonal trends. We also show how LLMs can naturally handle missing data without imputation through non-numerical text, accommodate textual side information, and answer questions to help explain predictions. While we find that increasing model size generally improves performance on time series, we show GPT-4 can perform worse than GPT-3 because of how it tokenizes numbers, and poor uncertainty calibration, which is likely the result of alignment interventions such as RLHF.

1 Introduction

LLMTIME reframes forecasting as next-token prediction by encoding numerical time series as text for pretrained LLMs. It achieves strong zero-shot forecasting while leveraging LLM biases, flexible uncertainty representations, and auxiliary capabilities.

  • Method: LLMTIME encodes time series as numerical strings and forecasts by sampling text completions from pretrained LLMs.The method applies pretrained models without downstream training.
  • Zero-shot performance: LLMTIME can match or exceed purpose-built time series methods across diverse problems without fine-tuning on target datasets.This zero-shot setting avoids specialized fine-tuning procedures and supports limited-data scenarios.
  • Why it works: LLM simplicity and repetition biases align with salient time-series structures such as seasonality.The paper investigates these biases as one explanation for forecasting performance.
  • Capabilities: LLMs naturally accommodate missing data, multimodal distributions, textual side information, and prediction-explanation questions.These capabilities extend forecasting beyond point prediction.
  • Scaling and limitations: Performance generally improves with model scale, but GPT-4 has worse uncertainty calibration and forecasting performance than GPT-3.The paper links this exception to alignment interventions such as RLHF and altered numerical tokenization.

2 Background

Time-series forecasting shares sequence structure with language modeling but adds heterogeneous scales, sampling rates, missing values, and substantial uncertainty. The paper positions zero-shot numerical forecasting as distinct from prior fine-tuned language-model approaches, with tokenization and evaluation as central challenges.

  • Numerical representation: Language models represent probability distributions over sequences, but BPE tokenization can break numbers into awkward chunks that hinder numerical pattern learning.Tokenization details can strongly affect forecasting behavior.
  • Evaluation: MAE ignores forecast uncertainty, whereas CRPS evaluates distributional quality for sampled forecasts but does not capture correlations between time steps.The paper adapts LLM likelihoods to continuous densities for model comparison.
  • Prior work and contribution: Prior language-model time-series methods commonly fine-tune encoders, while LLMTIME uses LLMs directly without fine-tuning or added prompt engineering.Its probabilistic formulation also targets uncertainty in highly stochastic series.
  • Tokenization: Added spaces help GPT-3 tokenize one digit per token but hurt LLaMA-2, whose tokenizer already assigns separate tokens to digits and spaces.The Australian Wine example shows that preprocessing effects depend on the base model.

3 LLMTIME: Forecasting with Language Models

LLMTIME preprocesses numerical series into token sequences, samples forecasts with standard autoregressive procedures, and converts discrete token probabilities into continuous densities. Its design emphasizes tokenization and expressive uncertainty modeling for multimodal and heavy-tailed data.

  • Forecasting pipeline: Correct preprocessing is essential because incorrect numerical handling can produce unusable forecasts.Forecasting otherwise follows standard language-model sampling after values are converted to strings.
  • Tokenization: GPT-3 benefits from added spaces that produce one token per digit, whereas the same modification hurts LLaMA models.For LLaMA, spaces add nuisance inputs and lengthen sequences without simplifying their structure.
  • Sampling: Forecasts use multiple sampled completions to construct point estimates such as medians or probabilistic forecasts such as quantiles.Temperature scaling, logit bias, and nucleus sampling control generation.
  • Continuous likelihoods: With n digits in base B, digit sequences define B^n bins of width B^-n, forming a hierarchical softmax over numbers.This discrete representation provides increasingly fine numerical resolution as precision increases.
  • Continuous likelihoods: Placing uniform densities within discrete bins converts token probabilities into continuous likelihoods over numerical values.A change-of-variables factor maps likelihoods back to the original input space.
  • Continuous likelihoods: The resulting construction efficiently represents flexible, high-resolution continuous distributions despite discrete number tokenization.The paper motivates this flexibility for sophisticated uncertainty modeling.
  • Flexible distributions: Autoregressive models can fit heavy-tailed and multimodal distributions that challenge simple Gaussian or Laplace observation models.The paper compares these fits with GMMs and fixed-resolution binning using Wasserstein distance.

4 Experiments

LLMTIME achieves strong zero-shot forecasting performance across deterministic and probabilistic benchmarks, including settings with diverse datasets and limited training data. Its probabilistic forecasts outperform dedicated baselines and PromptCast while retaining data efficiency.

  • Probabilistic results: LLMTIME generates higher likelihoods and CRPS values than dedicated models such as ARIMA, TCNs, and N-HiTS in zero-shot evaluation.It also performs well on deterministic metrics such as MAE.
  • Deterministic results: LLMTIME with GPT-3 or LLaMA-2 70B ranks best or second best on all considered deterministic benchmarks without trainable parameters.The benchmarks collectively contain 29 datasets from diverse sources, with varying lengths and noise levels.
  • Probabilistic results: On Darts, LLMTIME outperforms all baselines, including PromptCast, on probabilistic metrics such as NLL and CRPS.The comparison aggregates results over the Darts datasets.
  • Data efficiency: LLMTIME remains effective when given only a fraction of the original training data, while competing methods degrade rapidly as data decreases.This sample efficiency is particularly relevant to time-series settings characterized by relative data scarcity.
  • Comparison with PromptCast: LLMTIME significantly outperforms PromptCast on both CRPS and MAE aggregated over the Darts datasets.The methods differ in forecasting formulation, tokenization, and data rescaling; the paper identifies the latter strategy as crucial for good performance.

5 Origins of Zero-Shot Performance

The paper attributes zero-shot extrapolation to LLM preferences for simple, repetitive, and arithmetic patterns that match common time-series structure. These capabilities support generalization, but composing multiple patterns remains challenging for GPT-3.

  • Simplicity bias: LLMs favor simple generation rules, giving them an Occam’s-razor-like prior that can support zero-shot extrapolation.The paper contrasts simple rules with arbitrarily many more complex rules that fit the same observed sequence but generalize poorly.
  • Simplicity bias: GPT-3 likelihoods favor symbolic-regression solutions that balance training loss and complexity, leading to good generalization.Figure 6 compares these solutions with ARIMA and TCN models using predicted intervals and log likelihoods.
  • Repetition and periodicity: LLM repetition bias corresponds to identifying and extrapolating periodic structure in time series.The paper illustrates this with a repeating numerical sequence whose likely next value continues the period.
  • Arithmetic and trend components: LLM arithmetic abilities map onto extrapolation of linear and exponential trends, while sigmoid trends can be represented through related update rules.The paper gives additive, multiplicative, and sigmoid recurrence examples.
  • Compositional patterns: Composing multiple patterns is harder because the model must identify the composite rule and perform several operations within the same token budget.GPT-3 succeeds only sometimes, whereas GPT-4 performs compositions more consistently.

6 Special Properties of LLMs

LLMs extend time-series forecasting with scaling behavior and text-based handling of missing data and queries. However, alignment and tokenization choices can undermine forecasting performance.

  • Chat models: LLM forecasting performance appears negatively affected by alignment procedures, with LLaMA-2 chat models typically underperforming corresponding base models.The paper discusses instruction tuning and RLHF as examples of alignment procedures.
  • Base models and forecasting performance: Forecasting performance improves with reasoning performance across GPT-3, LLaMA, and LLaMA-2 base models.The study measures reasoning with MMLU accuracy and forecasting with NLL/D and CRPS on Darts.
  • Chat models: GPT-4 has significantly larger forecasting error than GPT-3 on the Darts datasets.Altered tokenization prevents easily forcing individual digits into an unbroken numerical stream, and API restrictions prevent likelihood evaluation.
  • Missing data: LLaMA-2 70B is more resilient to missing-value corruption and achieves CRPS competitive with methods using interpolation.Traditional-method likelihoods rapidly deteriorate under corruption, whereas LLMTIME represents missingness directly with text such as “NaN”.
  • Textual understanding: LLMs can answer textual questions about time series, including identifying the generating function with GPT-4 and chain-of-thought prompting.The paper reports that GPT-4’s accuracy is evaluated on synthetic time series, while examples show textual analysis of trend and periodicity.

7 Discussion

The discussion presents LLMs as pretrained time-series forecasters whose text-based training supports generalizable patterns and question answering. It also identifies context length and arithmetic as limitations for harder or multivariate problems.

  • Implications: Encoding numerical values as text enables pretrained LLMs to forecast time series without substantial computational resources, domain expertise, or many downstream training points.The discussion frames this as a broader unification of capabilities across tasks and modalities.
  • Limitations: LLM forecasters inherit limited context windows, making multivariate time-series problems a more significant challenge.The paper identifies extending context windows and combining those advances with forecasting as future research directions.
  • Limitations: Current LLM architectures may also struggle with arithmetic and recursive or compositional operations on particularly challenging time series.The paper notes that many time series may not require precise arithmetic, leaving the extent of this limitation unresolved.

A Detailed method and hyperparameters

LLMTIME combines time-series scaling, validation-based hyperparameter selection, and token-probability adjustments to produce continuous forecasts from language models. Its evaluation compares zero-shot GPT-3 forecasts with established forecasting baselines.

  • Input scaling: GPT-3 scales each series with an affine transformation whose digit thresholds are controlled by α and β.The transformation shifts by b and scales by the α-percentile of the shifted series; a basic scaling-only alternative is also considered.
  • Validation tuning: Validation likelihood selects hyperparameters using a held-out suffix without training LLMTIME.The validation series is the final T observations, or the final half when training data are shorter than 2T.
  • Likelihood adjustment: LLMTIME converts language-model token probabilities into continuous densities by restricting probability mass to tokens used by the numerical encoding.When logits are unavailable, the adjustment must be approximated from limited top-k log probabilities.
  • Evaluation: Figure 9 evaluates GPT-3 on short univariate series recorded after its training cutoff against popular time-series models using negative log likelihood.The figure compares predicted medians and 10–90th percentile intervals from in-context forecasting.
  • Likelihood adjustment: The adjustment estimates non-numerical token mass l0 and renormalizes the remaining probabilities to obtain adjusted probabilities.This approximation is needed for black-box APIs that do not expose the full discrete distribution.

B Addressing Memorization Concerns in GPT-3 Evaluations

The evaluation addresses possible GPT-3 memorization by discussing data leakage, testing post-cutoff series, and comparing against multiple forecasting baselines across selected datasets. The post-cutoff experiment finds GPT-3 competitive with or better than baselines using only in-context learning.

  • Memorization concern: Benchmark evaluations may overestimate generalization because GPT-3 can memorize datasets or encounter closely related data during pretraining.The concern is especially relevant when black-box models have unknown training data.
  • Memorization concern: Idiosyncratic formatting and limited dataset redundancy make memorization an unlikely explanation for GPT-3’s performance a priori.The paper notes that memorization increases with redundancy, while the evaluation formatting is unlikely to occur in training data.
  • Post-cutoff evaluation: The direct post-cutoff test uses Istanbul Traffic, TSMC Stock, and Turkey Power series recorded after September 2021.The datasets contain 267, 246, and 366 observations respectively, with dataset-specific sampling and aggregation choices.
  • Post-cutoff evaluation: GPT-3 competes with or surpasses baseline models on all post-cutoff tasks using in-context learning alone.Each series reserves its last 30 observations for testing, and methods are tuned over the same hyperparameter grid.
  • Benchmark design: The broader evaluation selects 19 Monash datasets subject to computational and context-window constraints.Aggregation normalizes MAE by a naive last-value predictor, emphasizing datasets where methods differ substantially from that baseline.

C.3 Informer datasets

The Informer-dataset evaluation compares LLMTIME with efficient-transformer baselines on five multivariate datasets and prediction horizons of 96 and 192. LLMTIME remains competitive in aggregate, although performance varies by dataset and declines slightly at the longer horizon.

  • Dataset and evaluation setup: The evaluation uses ETTm2, exchange_rate, electricity, traffic, and weather with prediction horizons of 96 and 192.Main-text results use horizon 96; a smaller evaluation set takes the final horizon timesteps from each series.
  • Probabilistic results: Figure 10 reports median predictions and negative log likelihoods for GPT-3 and LLaMA-2 70B across Darts datasets.The shaded regions represent the 10th–90th quantiles of sampled forecasts.
  • Baselines: Informer, Reformer, Autoformer, and FEDformer provide the efficient-transformer baselines.Their designs use sparse attention, locality-sensitive hashing, Fourier-space attention, or frequency-based decompositions.
  • LLMTIME configuration: LLMTIME uses LLaMA-2 70B with α = 0.99, β = 0.3, temperature = 1.0, nucleus = 0.9, and precision = 3.The long series are truncated to fit the 4096-token context window.
  • Probabilistic results: LLMTIME typically obtains better likelihoods than baselines and identifies trend and seasonal components, with only about 300 observations fitting in context after tokenization.Sunspots and HeartRate exceed that amount, with 705 and 900 observations respectively.
  • Longer-horizon results: At horizon 192, LLMTIME’s relative performance is slightly diminished, largely because of the electricity and traffic datasets.The overall trend remains similar to the shorter-horizon evaluation.

D Simplicity bias experiments

The simplicity-bias experiments test whether LLMTIME can model synthetic functions and extrapolate deterministic or compositional patterns. GPT-3 handles some compositions well, but Linear + Cosine remains difficult during sampling, while GPT-4 can behave erratically.

  • Synthetic setup: The synthetic experiment generates noisy observations from f(x) = cos(x) + x and fits symbolic expressions to the first 140 timesteps.PySR searches expressions using arithmetic, trigonometric, exponential, and square operators, ranking solutions by complexity.
  • Synthetic setup: The symbolic-regression outputs include increasingly complex expressions involving nested trigonometric functions and linear terms.Examples are ranked as solutions 3, 4, and 5 in the reported results.
  • GPT-3 results: GPT-3 shows good likelihood understanding of Linear + Cosine but has more trouble sampling it.The paper suggests that high-likelihood solutions may not be typical samples as a possible explanation for this discrepancy.
  • GPT-4 comparison: GPT-4 identifies and extrapolates deterministic synthetic patterns but sometimes behaves erratically.On real DARTS series, its extrapolations are plausible but worse than GPT-3, with poorer uncertainty calibration and high CRPS.

E GPT-4

GPT-4 performs better on synthetic benchmarks but worse than GPT-3 on nondeterministic DARTS data, with less well-calibrated uncertainty. Tokenization and interface constraints complicate its application to numerical forecasting.

  • Naive number tokenization and system–user separation require prompting strategies to obtain numerical continuation from GPT-4.Feeding numbers without additional spaces addresses tokenizer limitations, while extra interface tokens otherwise elicit discussion instead of continuation.
  • GPT-4 is considerably better performing on the synthetic benchmarks, although numerical decoding sometimes fails before the full output.
  • GPT-4 predictions on DARTS datasets are slightly worse than GPT-3 for nondeterministic time series.
  • GPT-4 uncertainties are much less well calibrated than GPT-3’s on the DARTS datasets.

F Multimodal Text Understanding of Time Series

GPT-4 is prompted to infer which candidate function generated a numerical time series, using textual observations and reasoning before producing an answer. The examples show correct classification for linear, exponential, and square series, while some functions remain difficult.

  • The experiment asks GPT-4 to identify the function generating a synthetic time series from code and numerical values in a zero-shot setting.
  • Guiding prompts before and after the numerical series instruct GPT-4 to analyze the data, compare candidate functions, and provide a named answer.The prompt requests observations, step-by-step reasoning, and an answer in the form “Answer: function_name.”
  • Across 20 trials, x_times_sine, beat, and sinc were predicted incorrectly 100% of the time.
  • The model’s textual analysis uses trend, periodicity, smoothness, symmetry, and acceleration to distinguish candidate functions.Examples contrast linear increase without oscillation, accelerating growth, and symmetric parabolic structure against mismatched candidates.
  • GPT-4 correctly classified the linear, exponential, and square synthetic time series in the illustrated examples.The sample outputs identify “linear,” “exp,” and “square” as the generated functions.
Loading 2310.07820v3…