PaperlayerView exact revision on arXiv

Source-linked AI summary

LLM4TS: Aligning Pre-Trained LLMs as Data-Efficient Time-Series Forecasters

Ching Chang, Wei-Yao Wang, Wen-Chih Peng, Tien-Fu Chen

arXiv:2308.08469v6cs.LG

TL;DR

Multivariate time-series forecasting is important but difficult with limited data because LLMs adapt poorly to time-series patterns and multi-scale temporal information. LLM4TS aligns a pre-trained LLM through two-stage fine-tuning and two-level temporal aggregation, outperforming existing methods across seven datasets in full- and few-shot settings.

  • Problem

    Limited time-series data and LLMs’ weak adaptation to time-series patterns and multi-scale temporal information create a forecasting challenge.

  • Method

    LLM4TS performs time-series alignment followed by forecasting fine-tuning and uses two-level aggregation to embed multi-scale temporal information into patched data.

  • Results

    Across 7 datasets, LLM4TS outperforms state-of-the-art methods, including train-from-scratch models, in full- and few-shot scenarios.

  • Takeaways & Limitations

    LLM4TS provides data-efficient forecasting performance while retaining most pre-trained LLM parameters.

  • Takeaways & Limitations

    The study primarily evaluates GPT-2 and forecasting, leaving newer LLMs and tasks such as classification and anomaly detection for future work.

Abstract

from arXiv · show

Multivariate time-series forecasting is vital in various domains, e.g., economic planning and weather prediction. Deep train-from-scratch models have exhibited effective performance yet require large amounts of data, which limits real-world applicability. Recently, researchers have leveraged the representation learning transferability of pre-trained Large Language Models (LLMs) to handle limited non-linguistic datasets effectively. However, incorporating LLMs with time-series data presents challenges of limited adaptation due to different compositions between time-series and linguistic data, and the inability to process multi-scale temporal information. To tackle these challenges, we propose LLM4TS, a framework for time-series forecasting with pre-trained LLMs. LLM4TS consists of a two-stage fine-tuning strategy: the time-series alignment stage to align LLMs with the nuances of time-series data, and the forecasting fine-tuning stage for downstream time-series forecasting tasks. Furthermore, our framework features a novel two-level aggregation method that integrates multi-scale temporal data within pre-trained LLMs, enhancing their ability to interpret time-specific information. In experiments across 7 time-series forecasting datasets, LLM4TS is superior to existing state-of-the-art methods compared with trained-from-scratch models in full-shot scenarios, and also achieves the highest rank in few-shot scenarios. In addition, evaluations compared with different unsupervised representation learning approaches highlight LLM4TS's effectiveness with representation learning in forecasting tasks. Ablation studies further validate each component's contribution to LLM4TS and underscore the essential role of utilizing LLM's pre-trained weights for optimal performance. The code is available at https://github.com/blacksnail789521/LLM4TS.

1 Introduction

LLM4TS addresses limited data and two adaptation challenges in applying pre-trained LLMs to multivariate time-series forecasting. It aligns LLMs to time-series data, integrates multi-scale temporal information, and achieves strong full- and few-shot performance.

  • Limited large-scale time-series data constrains train-from-scratch representation learners, especially when industrial sensor data cannot be combined across product types.
  • LLMs face limited adaptation to time-series patterns because they are pre-trained primarily on linguistic corpora.
  • LLMs struggle to process multi-scale temporal information, including time units and dates needed to identify forecasting patterns.
  • LLM4TS uses time-series alignment followed by forecasting fine-tuning, while freezing most pre-trained LLM parameters.
  • A two-level aggregation strategy embeds multi-scale temporal context into patched time-series data.
  • 7 real-world benchmarks show LLM4TS outperforming state-of-the-art methods, including train-from-scratch models, while 5% data surpasses the best baseline using 10%.

2 Related Work

Prior work spans transfer learning with LLMs, train-from-scratch forecasting, and self-supervised time-series representation learning. LLM4TS is positioned as a strong alternative in both full- and few-shot settings.

  • LLMs transfer across images, audio, tabular data, and time-series data, often retaining most parameters to preserve data-independent representation learning.
  • Train-from-scratch forecasting models perform strongly with sufficient data but lose efficacy in limited-data scenarios.
  • DLinear showed that a single-layer linear model can surpass many sophisticated Transformer-based forecasting approaches.
  • Time-series self-supervised learning commonly uses CNN or RNN backbones, while Transformers offer long-range dependency modeling suited to complex sequential relationships.

3 Problem Formulation

The forecasting problem uses sliding windows over complete, evenly sampled multivariate series. Past observations are used to predict a specified future horizon across all features.

  • A stride-1 sliding window extracts sequential samples from the multivariate time series.
  • Each window contains Tin past steps and Tout future steps, with total length Tin + Tout.
  • At each time step, dt is a C-dimensional feature vector, and xin ∈ R^Tin×C predicts xout ∈ R^Tout×C.

4 The Proposed LLM4TS

LLM4TS adapts a pre-trained LLM to multivariate time series through alignment, temporal encoding, and forecasting fine-tuning. Its design replaces text-oriented input handling with patched time-series representations and multi-scale temporal embeddings while preserving most pre-trained parameters.

  • Framework: LLM4TS first aligns the pre-trained GPT-2 backbone with patched time-series data using an autoregressive objective, then transfers the resulting weights to forecasting fine-tuning.The alignment stage predicts a sequence shifted one patch to the right, matching the causal language model’s pre-training methodology.
  • Time-Series Tokenization: Channel-independence and patching convert multivariate sequences into univariate patch tokens, reducing the time dimension from T_in to T_p.Patching also expands the feature dimension from 1 to the patch length P and reduces Transformer time and space complexity quadratically.
  • Three Encodings for Patched Time-Series Data: A convolutional token encoder replaces the text token encoder because patched time-series inputs are vectors and convolution preserves local semantic information.The encoder produces token embeddings e_token ∈ R^(T_p×D).
  • Three Encodings for Patched Time-Series Data: Token, positional, and temporal embeddings are summed before entering the pre-trained Transformer blocks.Positional embeddings come from a trainable lookup table over patch locations.
  • Forecasting Fine-tuning: Most Transformer parameters remain frozen, while Layer Normalization Tuning and LoRA train only 1.5% of the pre-trained LLM’s parameters.Forecasting fine-tuning uses linear probing followed by full fine-tuning, and incorporates RevIN to address distribution shifts between training and testing data.

5 Experiments

The experiments evaluate LLM4TS on real-world multivariate forecasting benchmarks using standard error metrics and comparisons with forecasting and representation-learning baselines.

  • Datasets: Experiments use 7 real-world, publicly accessible benchmark datasets for long-term forecasting.Dataset statistics include feature counts, total lengths, and sampling frequencies.
  • Datasets: Weather, Traffic, and Electricity represent diverse forecasting settings involving climatological, road-occupancy, and customer power-usage data.Weather covers approximately 1,600 U.S. locations; Traffic contains hourly freeway-sensor observations; Electricity contains hourly usage from 321 customers.
  • Evaluation Metrics: Forecasting performance is evaluated with Mean Squared Error (MSE) and Mean Absolute Error (MAE).MSE and MAE compare predicted future data with the corresponding actual future data across samples.
  • Baselines: The forecasting comparison includes state-of-the-art models such as GPT4TS and DLinear, while representation-learning comparisons include PatchTST, BTSF, TS2Vec, TNC, and TS-TCC.GPT4TS uses patching, channel independence, and a pre-trained GPT-2; the representation-learning baselines learn from unlabeled time-series data.
  • Implementation Details: Experiments cover few-shot learning, full-shot forecasting, and ablation studies under settings adapted from PatchTST for consistent comparison.The look-back window is 336 or 512, and the patch length is 16.

5.1 Few-Shot Learning in Long-Term Time-Series Forecasting

Few-shot experiments assess forecasting with 5% and 10% of training data across multiple datasets and prediction lengths. LLM4TS generally outperforms competing pre-trained and train-from-scratch approaches in these limited-data settings.

  • 5% Training Data: With 5% of training data, LLM4TS outperforms the best 10%-data baseline and leads across 5 of 6 reported evaluations.The 5%-data and 10%-data experiments preserve consistent train, validation, and test splits.
  • 10% Training Data: With 10% of training data, LLM4TS leads in 5 of 8 evaluations on the largest Traffic dataset.PatchTST leads in the full-shot Traffic setting, but this advantage does not extend consistently to few-shot scenarios.
  • Comparison with GPT4TS: LLM4TS outperforms GPT4TS across all datasets after adding time-series alignment and multi-scale temporal information integration.These components distinguish LLM4TS from the related pre-trained-LLM baseline in the reported few-shot comparisons.
  • Comparison with LLM-based Methods: Against Time-LLM, TEMPO, and TEST, LLM4TS achieves superior overall performance in both 5% and 10% training-data scenarios.TEMPO is evaluated zero-shot because it is a prompt-based method; it performs best on ETTh1 and ETTh2, while Time-LLM leads on ETTm1 and ETTm2.

5.2 Full-Shot Learning in Long-Term Time-Series Forecasting

Full-shot experiments average long-term forecasting results over prediction lengths from 96 to 720. LLM4TS outperforms deep train-from-scratch methods overall, while PatchTST remains strongest on Traffic and Time-LLM is competitive among LLM-based methods.

  • Overall Full-Shot Results: LLM4TS outperforms all deep train-from-scratch methods with full dataset access in the reported long-term forecasting comparison.The authors attribute this result to two-stage fine-tuning and integration of multi-scale temporal information.
  • Dataset-specific Comparison: PatchTST outcompetes both LLM4TS and GPT4TS on the largest dataset, Traffic, in the full-shot scenario.The authors note that traditional train-from-scratch models may sometimes excel when complete data access and sufficient data volume are available.
  • Comparison with LLM-based Methods: Time-LLM is the leading model among the compared LLM-based methods, while LLM4TS remains close in performance.Time-LLM reprograms LLMs by converting time series into text prototypes and using prompts to guide predictions.
  • Comparison with LLM-based Methods: LLM4TS outperforms Time-LLM in few-shot scenarios, demonstrating stronger performance under limited-data conditions.The full-shot comparison identifies Time-LLM as the leading LLM-based model, whereas the few-shot comparison favors LLM4TS.

5.3 Unsupervised Representation Learning

The study evaluates LLM4TS as an unsupervised representation learner by freezing a backbone pretrained on an autoregressive pretext task and training a linear downstream layer.

  • Evaluation Design: LLM4TS’s autoregressive time-series alignment objective is evaluated as an unsupervised representation-learning pretext task.The evaluation uses linear probing for time-series forecasting.
  • Linear Evaluation: The backbone is pretrained with the pretext task, its weights are frozen, and an attached linear layer is trained for downstream forecasting.This isolates the forecasting utility of the learned representations.

5.4 Ablation Study

The ablations show that time-series alignment, multi-scale temporal encoding, PEFT, LP-FT, and frozen pre-trained weights each contribute to LLM4TS performance, especially in few-shot forecasting.

  • Key Components in LLM4TS: 6.2% lower MSE is achieved on average in few-shot ETTh1 forecasting when the key LLM4TS components are incorporated.The study evaluates time-series alignment, multi-scale temporal encoding, and PEFT under full- and few-shot settings with 10% training data.
  • Key Components in LLM4TS: Few-shot scenarios gain more than full-shot scenarios from the main components, supporting LLM4TS’s data-efficient forecasting capability.The comparison is conducted under both full- and few-shot learning with 10% training data.
  • Key Components in LLM4TS: LoRA is more beneficial than Layer Normalization among the two PEFT methods in both full-shot and few-shot settings.The advantage is reported consistently across both learning scenarios.
  • Training Strategies: 0.7% and 2.51% average MSE improvements result from LP-FT in full-shot and few-shot ETTh1 learning, respectively.LP-FT is reported to outperform both linear probing and full fine-tuning in in-distribution and out-of-distribution scenarios.
  • Pre-trained Weights: 17.78% average MSE improvement over No Freeze is obtained when most pre-trained parameters remain frozen across Weather, ETTm1, and ETTm2.The ablation uses few-shot learning with 10% and 5% training data; No Freeze uses pre-trained weights without freezing layers.

5.5 Training and Inference Cost

LLM4TS reduces the number of trainable parameters through PEFT while its training and inference costs are compared with PatchTST and FEDformer under standardized conditions.

  • Training Parameters: 4% of LLM4TS’s parameters are trainable because most pre-trained parameters remain fixed and PEFT methods are used.The methods are Layer Normalization Tuning and LoRA.
  • Training and Inference Time: Training and inference times are compared with PatchTST and FEDformer on an NVIDIA Tesla V100 GPU using one batch.The comparison uses batch size 128, hidden dimension 768, and prediction length Tout = 96 on ETTh2.
  • Training and Inference Time: LLM4TS reports training time for both fine-tuning stages because its pipeline includes two-stage training.The evaluation is conducted under standardized batch-size and hidden-dimension settings.

6 Conclusion

LLM4TS adapts pre-trained LLMs for time-series forecasting through two-stage fine-tuning and multi-scale temporal aggregation. Across seven datasets, it outperforms existing state-of-the-art and train-from-scratch methods in full- and few-shot settings, while future work targets newer LLMs and additional tasks.

  • Conclusion: LLM4TS aligns pre-trained LLMs with time-series characteristics before fine-tuning them for forecasting tasks.Its two stages are time-series alignment and forecasting fine-tuning.
  • Conclusion: A novel two-level aggregation method integrates multi-scale temporal data within pre-trained LLMs to improve interpretation of time-related information.The method is presented as part of the LLM4TS framework.
  • Conclusion: Across 7 time-series forecasting datasets, LLM4TS outperforms existing state-of-the-art methods, including train-from-scratch models, in full- and few-shot scenarios.The conclusion reports this performance pattern across both data regimes.
  • Future Work: Future work will evaluate newer LLMs such as GPT-3.5 and LLaMA-2 and extend LLM4TS to classification and anomaly detection.GPT-2 was used as the primary LLM in this paper for fair comparison with GPT4TS.
Loading 2308.08469v6…