PaperlayerView exact revision on arXiv

Source-linked AI summary

Are Language Models Actually Useful for Time Series Forecasting?

Mingtian Tan, Mike A. Merrill, Vinayak Gupta, Tim Althoff, Thomas Hartvigsen

arXiv:2406.16964v2cs.LGcs.AI

TL;DR

The paper examines whether LLMs provide genuine benefits for traditional time-series forecasting. It performs ablations of three recent LLM-based methods across standard datasets and finds that simpler alternatives generally match or improve their forecasting performance at much lower computational cost, while pretraining, sequence transfer, and few-shot use provide no observed benefit.

  • Problem

    It asks whether language modeling is genuinely beneficial for traditional time-series forecasting, where direct connections between language modeling and forecasting remain largely undefined.

  • Method

    The paper performs three ablations of three recent LLM-based forecasting methods across thirteen standard datasets, replacing or removing LLM components and testing pretraining, sequence modeling, few-shot settings, and encoders.

  • Results

    LLM-free or simpler alternatives maintain or improve forecasting performance, reduce training and inference time by up to three orders of magnitude, and show no benefit from pretraining, sequence transfer, or few-shot use.

  • Takeaways & Limitations

    For established forecasting tasks, the paper suggests focusing LLM use on applications requiring textual reasoning, such as multimodal time-series tasks, reasoning, or social understanding.

  • Takeaways & Limitations

    The evaluation focuses on time-series forecasting, while the authors note that LLM abilities should also be assessed on tasks such as classification and question-answering.

Abstract

from arXiv · show

Large language models (LLMs) are being applied to time series forecasting. But are language models actually useful for time series? In a series of ablation studies on three recent and popular LLM-based time series forecasting methods, we find that removing the LLM component or replacing it with a basic attention layer does not degrade forecasting performance -- in most cases, the results even improve! We also find that despite their significant computational cost, pretrained LLMs do no better than models trained from scratch, do not represent the sequential dependencies in time series, and do not assist in few-shot settings. Additionally, we explore time series encoders and find that patching and attention structures perform similarly to LLM-based forecasters.

1 Introduction

This section asks whether LLMs genuinely benefit traditional time-series forecasting and tests that question through systematic ablations of recent LLM-based forecasters. Across standard datasets, simpler LLM-free alternatives generally match or outperform the original methods while using substantially less computation.

  • Scope of the claim: The paper distinguishes its findings from a claim that LLMs can never help time-series analysis, highlighting applications involving time-series reasoning, social understanding, and financial reasoning.Its stated focus is established forecasting tasks rather than multimodal or broader language–time-series applications.
  • Evaluation design: The study evaluates three popular LLM-based forecasting methods through three ablations on eight standard benchmark datasets, with additional evaluation on five MONASH datasets.The authors first reproduce the reference results, then compare the original methods with simplified alternatives.
  • Main findings: Replacing or removing the language model with attention layers, transformer blocks, random initialization, or no LLM yields comparable or better forecasting performance.The same pattern is observed on five datasets not studied by the reference methods.
  • Computational cost: Simpler ablations reduce training and inference time by up to three orders of magnitude while maintaining comparable performance.This creates a large computational advantage over the evaluated LLM-based forecasters.
  • Sequence modeling and few-shot settings: Randomly shuffling input time series produces no appreciable performance change, providing no evidence that the LLMs transfer sequence-modeling abilities from text to time series.The models also do not improve forecasting when trained with 10% of the training data.
  • Encoder analysis: A simple linear model using patching and attention as its encoder achieves forecasting performance similar to that of LLM-based forecasters.This result suggests that the encoder structure can account for much of the observed performance.

2 Related Work

Prior work applies LLMs to forecasting by adapting pretrained language models and encoding time series into discrete tokens or patches. Related research also develops smaller, efficient forecasters that can outperform bulkier architectures.

  • LLM-based forecasting: Recent LLM-based forecasting methods adapt pretrained models such as GPT-2 and use alignment or fine-tuning strategies for time-series inputs.The surveyed methods include approaches that fine-tune transformer modules and positional encodings.
  • Time-series encoders: LLM time-series systems encode inputs through discrete tokens, including overlapping patches that shorten sequences while retaining their features.Other encoder approaches decompose time series into components such as trend and seasonality.
  • Small and efficient forecasters: Smaller forecasting models can outperform bulkier counterparts; examples include DLinear and a lightweight model with only 10k parameters.DLinear combines decomposition techniques, while the lightweight model captures amplitude and phase information.

3 Experimental Setup

The experiments compare three recent LLM-based forecasters and controlled LLM ablations across standard time-series datasets using MAE and MSE. The setup preserves the reference methods’ remaining trainable structures while varying or removing the language-model component.

  • Reference methods: Three recent LLM-based forecasting methods use GPT2 or LLaMA with different alignment and fine-tuning strategies.The methods are OneFitsAll, Time-LLM, and CALF.
  • Ablation methods: The ablations remove the LLM, replace it with a single randomly initialized multi-head attention layer, or replace it with a single randomly initialized transformer block.These variants are named w/o LLM, LLM2Attn, and LLM2Trsf.
  • Ablation methods: The ablations retain the reference forecasters’ remaining trainable structure and pass encoded inputs directly to the output projection or through the replacement module.The unchanged portions remain trainable alongside the attention or transformer replacement.
  • Datasets: The evaluation uses ETT, Illness, Weather, Traffic, and Electricity real-world datasets with dataset-specific train-validation-test splits.ETT subsets contain hourly or 15-minute transformer recordings, while the other datasets cover illness, climate, traffic, and electricity consumption.
  • Evaluation: The figures compare LLM and ablation variants of OneFitsAll, CALF, and Time-LLM on ETTh1, ETTm2, and Electricity, with dashed lines showing original-work results.The comparison includes w/o LLM, LLM2Attn, and LLM2Trsf; additional MSE and dataset results appear in the appendix figures.
  • Evaluation: Performance is reported using mean absolute error (MAE) and mean squared error (MSE) between predicted and true time-series values.The evaluation formulas use D test-set elements and distinguish true values from predictions at each time-series index.

4 Results

Across thirteen datasets, simpler ablations generally match or outperform LLM-based forecasters, while requiring far less computation. Pretraining, sequence modeling, and few-shot benefits are not supported, whereas patching with attention can achieve comparable performance.

  • Forecasting performance: Ablations outperformed Time-LLM in 26/26 cases, CALF in 22/26 cases, and OneFitsAll in 19/26 cases across 13 datasets and two metrics.Across all prediction lengths, ablations won 35/40, 31/40, and 29/40 cases against the same methods, respectively, by MAE.
  • Computational cost: 2.17 minutes versus 3003 minutes: ablation methods trained much faster than Time-LLM on Weather while using 0.245 M versus 6642 M parameters.Time-LLM’s parameters were 6642 M, compared with 0.245 M for the ablations.
  • Pretraining: Random Initialization + Finetune achieved the best result 8 times, compared with 3 times for Pretraining + Finetune across 8 datasets and two metrics.Pretraining therefore provided very limited help for forecasting in this comparison.
  • Sequential dependencies: LLM-based methods were no more vulnerable to shuffled inputs than their ablations, providing no evidence that LLMs uniquely represent sequential dependencies in time series.The study tested whole-sequence, first-half, and swapped-half shuffling.
  • Few-shot learning: With 10% of each dataset, LLaMA and its no-LLM ablation each performed better in 8 cases, while CALF ablations could outperform the LLM.These experiments found no meaningful few-shot advantage for LLMs.
  • Time series encoders: PAttn, combining patching with one-layer attention, performed better than most encoders on small datasets and was comparable to LLM methods.For larger datasets, LTrsf performed better; overall, patching, attention, and transformer blocks were effective encoding choices.

5 Conclusion

The paper concludes that LLMs do not meaningfully improve time series forecasting, while simple ablations maintain or improve performance at considerably lower computational cost. It recommends greater focus on tasks at the interface of time series and language.

  • LLMs do not appear to meaningfully improve time series forecasting despite their recent popularity.
  • Simple ablations maintain or improve the performance of LLM-based forecasters while requiring considerably less compute.
  • The paper encourages greater focus on tasks connecting time series and language, including time series reasoning and social understanding.

A Limitations

The evaluation is limited to forecasting on uniformly sampled time-series datasets. A broader assessment should include other downstream tasks and non-uniform sequences.

  • The study evaluates LLMs for time-series forecasting but not other downstream tasks such as classification and question-answering.
  • The evaluation covers sequences with even time intervals, leaving forecasting on non-uniform sequences such as payment records and online purchases for future work.
  • The findings are intended to help researchers assess whether LLM components are necessary and whether their computational costs are reasonable for specific settings.

D Additional Experimental Details

The experiments use the same device for each reference method and its architectural modifications, while retaining reference training details except for selected adjustments in ablations.

  • Each reference method and architecture modification is trained and evaluated on the same device.
  • Time-LLM with LLaMA-7B uses an NVIDIA A100 GPU with 80GB memory, while GPT-2 methods use an NVIDIA RTX A6000 GPU with 48GB memory.
  • The LLM-free baselines can be trained on smaller GPUs, although memory-footprint analysis is outside the study’s scope.
  • Reproductions use original hyperparameters and model structures, while some ablations adjust learning rates or batch sizes because they have fewer parameters.

D.3 Details of Encoder Exploration and Simple Methods

The encoder exploration tests lightweight alternatives that encode time series before a linear forecast projection. Patching and attention-based models achieve performance comparable to or better than LLM-based methods.

  • Encoder exploration: The encoder experiments encode time series data and use a linear layer to project the resulting embeddings to forecasts.
  • Encoder exploration: PAttn combines patching and attention without positional embeddings or a feed-forward network, replacing the Transformer Encoder with a single-layer attention structure.
  • Simple methods: Across the results, PAttn appeared 34 times among the best and second-best performers, compared with 33 appearances for all LLM-based models combined.
  • Simple methods: LTrsf uses CALF’s encoder without cross-modal attention, while D-LTrsf and D-PAttn decompose time series into three subsequences before combining forecasts.

E Additional Experiments

Additional experiments are presented to further examine how well LLMs model time-series data.

  • Additional experiments evaluate the ability of LLMs to model time-series data.

E.1 Confidence Intervals for Forecasting

The paper reports confidence intervals for baseline forecasts and evaluates ablation generality on five additional datasets using specified prediction lengths. Across the reported results, prediction variation is described as quite small.

  • Confidence Intervals for Forecasting: Confidence intervals are reported for MAE and MSE predictions from Time-LLM, CALF, and OneFitsAll.The intervals are provided in Tables 9–14.
  • Confidence Intervals for Forecasting: The reported confidence-interval ranges are quite small across the results.
  • Confidence Intervals for Forecasting: Five additional datasets evaluate whether the paper’s ablations generalize beyond the reference methods’ studied datasets.The additional datasets are used in many time-series forecasting studies.
  • Confidence Intervals for Forecasting: Prediction lengths are 96, 192, 336, and 720 for Exchange Rate, and 30, 48, 56, and 12 for the other four additional datasets.

E.2 Complete Results

Complete results cover inference time, confidence intervals, randomized and shuffled inputs, and comparisons between LLM-based, non-LLM, and simplified ablation methods. Removing or replacing the LLM generally preserves or improves forecasting performance while reducing inference time, and the reported comparisons indicate the main finding is unchanged.

  • Confidence Intervals: The complete-results appendix includes confidence-interval tables for Time-LLM, CALF, and OneFitsAll across MAE and MSE predictions.
  • Complete Results: Additional tables report ablation results on datasets not studied by the reference methods and comparisons with original reference-paper results.
  • Ablation Results: Ablation studies show that removing the LLM or replacing it with single-layer attention or a Transformer does not reduce MAE performance and often improves it.The comparisons use original GPT-2 or LLaMA-based methods as references across several datasets.
  • Ablation Results: Ablation studies show that removing the LLM or replacing it with single-layer attention or a Transformer does not reduce MSE performance and often improves it.The vertical dashed line marks the original-paper results.
  • Comparisons: LLM-based methods slightly outperform non-LLM methods, but the ablations indicate this advantage is not due to the LLM.
  • Pretraining: Pretraining on language datasets is not necessary for time-series forecasting tasks.The comparison includes pretrained versus non-pretrained models and fine-tuning versus no fine-tuning.
  • Input Shuffling and Masking: Input shuffling or masking has no significantly different effect on forecasting degradation before versus after model modifications.This pattern is reported for ETTh1, Illness, ETTh2, Electricity, ETTm1, ETTm2, Weather, and Traffic.
  • Inference Times: Ablation methods use less inference time while providing better forecasting performance in most cases.Inference time and prediction accuracy are averaged across prediction lengths for the listed datasets.
Loading 2406.16964v2…