PaperlayerView exact revision on arXiv

Source-linked AI summary

PromptCast: A New Prompt-based Learning Paradigm for Time Series Forecasting

Hao Xue, Flora D. Salim

arXiv:2210.08964v5stat.MEcs.AIcs.CLcs.LGmath.ST

TL;DR

The paper asks whether language foundation models can be adapted to time-series forecasting beyond conventional numerical forecasting methods. It introduces PromptCast, which converts numerical sequences into prompts, and PISA, a large-scale dataset and benchmark; experiments report good forecasting performance and generalization ability for language models, with limitations in dataset scope and template-based prompting.

  • Problem

    The paper investigates whether large-scale pre-trained foundation models can be adapted for time-series forecasting, which has largely remained separate from NLP and computer vision foundation-model research.

  • Method

    PromptCast converts numerical historical observations and forecasting targets into natural-language input and output prompts, while PISA supports benchmarking numerical and language generation models.

  • Results

    Language models achieve good forecasting performance and generalization ability in PromptCast, with benchmark results covering multiple forecasting settings.

  • Takeaways & Limitations

    PromptCast with language generation models is presented as a promising research direction for time-series forecasting.

  • Takeaways & Limitations

    PISA mainly focuses on univariate forecasting, and fixed template-based prompting can limit prompt diversity and introduce template biases.

Abstract

from arXiv · show

This paper presents a new perspective on time series forecasting. In existing time series forecasting methods, the models take a sequence of numerical values as input and yield numerical values as output. The existing SOTA models are largely based on the Transformer architecture, modified with multiple encoding mechanisms to incorporate the context and semantics around the historical data. Inspired by the successes of pre-trained language foundation models, we pose a question about whether these models can also be adapted to solve time-series forecasting. Thus, we propose a new forecasting paradigm: prompt-based time series forecasting (PromptCast). In this novel task, the numerical input and output are transformed into prompts and the forecasting task is framed in a sentence-to-sentence manner, making it possible to directly apply language models for forecasting purposes. To support and facilitate the research of this task, we also present a large-scale dataset (PISA) that includes three real-world forecasting scenarios. We evaluate different SOTA numerical-based forecasting methods and language generation models. The benchmark results with various forecasting settings demonstrate the proposed PromptCast with language generation models is a promising research direction. Additionally, in comparison to conventional numerical-based forecasting, PromptCast shows a much better generalization ability under the zero-shot setting.

I. INTRODUCTION

PromptCast reframes time-series forecasting as sentence-to-sentence language generation, allowing language models to forecast transformed numerical sequences. The paper introduces PISA and benchmarks numerical and language-based methods across forecasting settings.

  • I. INTRODUCTION: PromptCast transforms numerical forecasting inputs and outputs into natural-language sentences, enabling language generation models to perform forecasting.Conventional methods map numerical sequences to numerical predictions, whereas PromptCast uses a sentence-to-sentence formulation.
  • I. INTRODUCTION: The paradigm offers a code-less alternative to increasingly complex forecasting architectures and may be more accessible to nonresearcher users.The paper contrasts PromptCast with Transformer-based models and their parameter-searching and training requirements.
  • I. INTRODUCTION: PISA is introduced as a large-scale dataset for prompt-based forecasting, covering weather temperature, energy consumption, and customer-flow scenarios.The dataset contains 311,932 instances across diverse time-series forecasting scenarios.
  • I. INTRODUCTION: The benchmark evaluates numerical forecasting methods and language generation models under train-from-scratch, zero-shot, multi-step, and multivariate settings.These settings are used to examine forecasting performance and generalization ability.
  • I. INTRODUCTION: PromptCast defines forecasting from historical numerical observations to future target values, with the numerical sequences converted into input and output prompts.The output prompt represents the forecasting target, while the input prompt represents the observed sequence.

A. Data Sources and Processing

PISA combines three real-world forecasting domains with template-based transformations from numerical sequences to paired input and output prompts. Its statistics span substantially different value distributions, supporting diverse benchmark conditions.

  • A. Data Sources and Processing: PISA includes weather, electricity-consumption, and human-mobility forecasting data from three real-world domains.The sources are City Temperature, Electricity Consumption Load, and SafeGraph Human Mobility Data.
  • A. Data Sources and Processing: The dataset construction preprocesses raw numerical data and transforms it into natural-language input and output formats with prompts.The PISA-numerical and PISA-prompt instances are aligned for comparison between numerical and language-based methods.
  • B. Template-Based Prompting: Templates pair historical observations and prediction-time indicators in the input prompt with the desired future value in the output prompt.The output prompt serves as the ground-truth label for training or evaluation and parallels source-target sentences in machine translation.
  • C. Statistics Overview: PISA contains 311,932 instances whose three subsets have distinct statistical characteristics and value distributions.The dataset is designed to represent diverse forecasting application domains and numerical ranges.
  • C. Statistics Overview: The CT subset includes negative values, ECL spans large values across a wide range, and SG contains relatively small values from 0 to 400.These distributions provide contrasting numerical regimes within the benchmark.

A. Evaluation Metrics

PromptCast evaluates forecasting by extracting numerical values from generated sentences, then applying standard error metrics alongside a Missing Rate for undecodable outputs. The benchmark compares language-generation and numerical methods under consistent implementation and temporal-embedding settings.

  • Evaluation Metrics: Missing Rate is defined as the percentage of test instances whose generated sentences do not yield a decodable numerical prediction.A lower Missing Rate indicates better performance.
  • Evaluation Metrics: RMSE and MAE evaluate extracted numerical predictions, with deep-learning results averaged across five random-seed runs.
  • Baselines: The benchmark tests 10 natural language generation models on PISA-prompt and conventional forecasting methods on PISA-numerical.Numerical baselines include naive, AutoARIMA, LSTM, TCN, Transformer, Informer, Autoformer, and FEDformer methods.
  • Implementation: Numerical-method implementations use official FEDformer repository defaults, with prediction length set to 7 and numerical data normalization included.
  • Implementation: Language models use EncoderDecoderModel or ConditionalGeneration implementations with HuggingFace fine-tuning and standard sequence-to-sequence training.The benchmark introduces no loss-function modifications or specific hyperparameter tuning.
  • Temporal Embeddings: Fixed temporal embeddings perform well across the numerical benchmark, leading on 5 of 6 metrics while timeF leads the remaining MAE metric on SG.These results support fixed embeddings as a favorable way to incorporate temporal cues.

2) Pre-trained Language Models:

Pre-trained language models achieve competitive forecasting performance across PISA subsets, while generated-output failures remain concentrated in a small set of models and CT cases. PromptCast also compares resource use, finding GPT-3.5 substantially more expensive than the evaluated PromptCast and numerical models.

  • Pre-training and Fine-tuning: The language models use pre-trained weights from general English corpora that do not contain the PISA time-series data, then are fine-tuned on each PISA training set.
  • Pre-trained Language Models: Bigbird achieves the best performance on 4 of 6 language-model metrics, while Bart and RoBERTa are also among the top performers.
  • Pre-trained Language Models: Language models perform reasonably on CT and ECL, improve ECL RMSE relatively substantially, and produce results comparable with numerical methods on SG.
  • Output Reliability: Less than 0.5% missing cases occur for ProphetNet, Electra, and BERT, all on the CT subset; unlisted methods have zero missing rate.Failure cases are potentially related to negative values that models fail to generate correctly.
  • Computation Cost: GPT-3.5 is substantially more computationally demanding and costly than the other evaluated models, making it less cost-effective.PromptCast language models show no substantial computation-time or cost increase versus Transformer-based numerical methods.

E. Training From Scratch

PromptCast remains usable without pretrained weights, though performance decreases; it also supports multi-step forecasting through prompt and horizon adjustments.

  • Training From Scratch: Pretrained weights improve performance across language-generation methods, but PromptCast remains robust when models are trained from scratch.Even without pretrained weights, especially Bart produces predictions comparable to numerical-based methods.
  • Zero-shot Performance: Zero-shot evaluation fine-tunes on two subsets and tests on the held-out third subset.The setting compares language models with Transformer-based numerical methods under transfer across the three subsets.
  • Zero-shot Performance: Except for Autoformer and FEDformer on ECL, numerical methods fail to produce satisfactory zero-shot predictions, whereas PromptCast generates reasonable forecasts.The paper links this result to strong generalization and identifies rapid deployment and cold-start forecasting as potential applications.
  • Prompt Ablation: Simplified basic and minimum prompts perform worse than the default template, while basic prompts outperform minimum prompts with the same language model.The comparison indicates that prompt design affects forecasting performance.
  • Multi-step Forecasting: PromptCast adapts to multi-step forecasting by revising the question and output templates while keeping the same language-model architectures.Longer prediction horizons generally increase missing rates; Pegasus remains close to Transformer-based methods in difficult seven-step settings.
  • Multi-step Forecasting: A 15-day observation period often improves forecasts of the same future length, likely because it captures weekly patterns.The paper also describes PromptCast as robust to dynamic observation and prediction lengths.

I. Multivariate Time Series Forecasting

PromptCast extends to multivariate forecasting by changing prompt templates while retaining the same language-foundation-model architectures. The pilot study finds prompt design affects performance, with unresolved challenges in representing feature relationships.

  • Multivariate Design: Multivariate PromptCast updates prompts to include multiple features while retaining the same prediction model structures and pretrained weights.Conventional numerical methods may require encoder updates and extra hyperparameter search when forecasting settings change.
  • Experimental Setup: The pilot study evaluates multivariate PromptCast on Beijing Air Quality Data using a chronological train, validation, and test split.The dataset covers 2013/03/01–2017/02/28 and is divided into training, validation, and test periods.
  • Prompt Design: Prompt A jointly describes three features in its output, Prompt B separates their values into three sentences, and Prompt C decomposes features into univariate prompts.These templates represent alternative ways to structure multivariate inputs and outputs.
  • Results: All three language models generate plausible future descriptions with almost all zero missing rates, but prompt design affects prediction performance.Prompt C performs better than Prompt A and Prompt B in the reported comparison.
  • Results: PromptCast achieves better MAE than a Transformer-fixed baseline in one comparison but worse RMSE than Transformer-based baselines.The authors attribute the weaker overall comparison to prompts that do not fully characterize relations among features.
  • Open Question and Future Work: Further research is needed to understand multivariate PromptCast and design prompts that represent internal correlations among temporal features.The paper identifies multivariate prompt design as an open question.

VI. DISCUSSION AND CONCLUSION

PromptCast applies language models to time-series forecasting through a new task, dataset, and benchmark. The paper reports good forecasting performance and generalization ability, while identifying dataset scope and interpretability as areas for future work.

  • PromptCast uses language models to predict time series in a language-generation manner.
  • PISA is a large-scale dataset covering weather temperature, energy consumption, and customer flow forecasting scenarios.
  • The benchmark evaluates numerical forecasting methods and language generation models across train-from-scratch, zero-shot, multi-step, and multivariate settings.
  • Further interpretability and explainability studies are proposed to clarify PromptCast models’ underlying mechanisms and limitations.
  • The released dataset mainly focuses on univariate time series, with multivariate coverage identified as future work.
  • Future work includes autoregressive multi-step forecasting, heterogeneous datasets, and extensions to classification and anomaly detection.

APPENDIX

The appendix documents supplementary analyses and dataset materials. It includes cost–accuracy plots, hosting and licensing information, and a dataset datasheet.

  • Section A contains scatter plots relating forecasting-method cost to accuracy.
  • Section B introduces hosting and licensing information for the PISA dataset.
  • Section C presents a Datasheet for Datasets for PISA.

A. Additional Cost Analysis

The additional cost analysis compares forecasting methods using plots of cost and accuracy across the three PISA subsets. PromptCast is described as accurate while remaining cost-effective.

  • Three plots represent the CT, ECL, and SG PISA subsets and compare forecasting-method cost with accuracy such as RMSE.
  • PromptCast presents good forecasting accuracy while maintaining cost-effectiveness in the plotted comparisons.

B. PISA Hosting and Licensing

PISA draws on three external data sources and is distributed with derived materials under specified access and licensing conditions. The repository provides the dataset, benchmark code, and generated language-model examples.

  • CT source data comes from the Average Daily Temperature Archive and is available for research and non-commercial purposes.
  • ECL uses a processed Informer-repository version of UCI data licensed under Apache License 2.0.
  • SG raw data is accessed through SafeGraph Data for Academics, whose terms govern access and privacy-related data handling.
  • Aggregated and derived data may be shared under SafeGraph terms, and PISA materials are available through the project repository.
  • The PISA dataset is distributed under CC BY-NC-SA 4.0.

C. PISA Datasheet

PISA is a self-contained dataset created to support PromptCast research, comprising prompt-based forecasting instances from three real-world scenarios. Its instances use input prompts as model inputs and output prompts as labels, with chronological train/validation/test splits.

  • PISA was created to support research on the novel PromptCast task.
  • The dataset covers city temperature, user electricity consumption, and POI visitor counts across weather, energy, and human mobility scenarios.
  • The dataset contains 311,932 instances in total.
  • Instances consist of separate input and output prompts, with the output prompt serving as the label.
  • Each subset is divided chronologically into train, validation, and test partitions at a 7:1:2 ratio.
  • The dataset is self-contained and reports no missing data beyond information intentionally omitted, such as SG POI geolocation.
Loading 2210.08964v5…