PaperlayerView exact revision on arXiv

Source-linked AI summary

TEMPO: Prompt-based Generative Pre-trained Transformer for Time Series Forecasting

Defu Cao, Furong Jia, Sercan O Arik, Tomas Pfister, Yixiang Zheng, Wen Ye, Yan Liu

arXiv:2310.04948v3cs.LGcs.CL

TL;DR

Time-series forecasting lacks a broadly transferable pretrained approach that adequately captures temporal structure and adapts across distributions. TEMPO combines trend-seasonal-residual decomposition with soft prompting in a GPT-based framework, outperforming existing baselines in zero-shot forecasting and extending to multimodal inputs. The paper presents this result as evidence of the potential for foundational time-series models.

  • Problem

    Time-series architectures vary across applications, while pretrained approaches have limited understanding of temporal characteristics and multimodal settings.

  • Method

    TEMPO combines STL-style trend, seasonality, and residual decomposition with adaptive soft prompts in a pretrained GPT backbone.

  • Results

    TEMPO surpasses existing baselines on average across prediction horizons and datasets, improving MAE by around 6.5% on weather and 19.1% on ETTm1 versus PatchTST.

  • Takeaways & Limitations

    TEMPO’s zero-shot and multimodal results highlight the potential of prompt-based foundational models for time-series forecasting.

  • Takeaways & Limitations

    The zero-shot configuration excludes the target dataset’s history and horizon values during training.

Abstract

from arXiv · show

The past decade has witnessed significant advances in time series modeling with deep learning. While achieving state-of-the-art results, the best-performing architectures vary highly across applications and domains. Meanwhile, for natural language processing, the Generative Pre-trained Transformer (GPT) has demonstrated impressive performance via training one general-purpose model across various textual datasets. It is intriguing to explore whether GPT-type architectures can be effective for time series, capturing the intrinsic dynamic attributes and leading to significant accuracy improvements. In this paper, we propose a novel framework, TEMPO, that can effectively learn time series representations. We focus on utilizing two essential inductive biases of the time series task for pre-trained models: (i) decomposition of the complex interaction between trend, seasonal and residual components; and (ii) introducing the design of prompts to facilitate distribution adaptation in different types of time series. TEMPO expands the capability for dynamically modeling real-world temporal phenomena from data within diverse domains. Our experiments demonstrate the superior performance of TEMPO over state-of-the-art methods on zero shot setting for a number of time series benchmark datasets. This performance gain is observed not only in scenarios involving previously unseen datasets but also in scenarios with multi-modal inputs. This compelling finding highlights TEMPO's potential to constitute a foundational model-building framework.

1 INTRODUCTION

Time-series models achieve strong results, but architectures vary across applications, motivating foundation-model approaches for transferable representation learning. TEMPO addresses this challenge by combining decomposition of temporal patterns with prompt-based adaptation.

  • Existing deep neural architectures achieve state-of-the-art time-series results, but their strongest designs vary across applications and domains.
  • Foundation models have advanced representation learning in NLP and CV, prompting exploration of pretrained models for time-series tasks.
  • Theoretical and practical understanding of general time-series transformers remains limited compared with generative models in other domains.
  • TEMPO decomposes inputs into trend, seasonality, and residual components before mapping them into embeddings for a generative pretrained transformer.
  • TEMPO combines temporal-pattern modeling with prompt-based learning to obtain more transferable representations across time-series data.

2 RELATED WORKS

Related work applies large language models and prompt tuning to time series, but often does not adequately represent temporal characteristics or support flexible multimodal extension. TEMPO builds on prompt-tuning ideas while explicitly incorporating temporal inductive biases.

  • Recent studies investigate pretrained language models for time-series analysis by leveraging and fine-tuning LLMs.
  • Existing LLM-based approaches often construct embeddings directly from time series without adequately capturing their inherent characteristics.
  • Multimodal time-series research remains limited, and METS is difficult to extend because its embedding alignment is modality-specific.
  • Prompt tuning adapts pretrained foundation models to downstream tasks using trainable soft prompts, prefix prompts, or other prompt designs.
  • This work explores prompt design as an inductive bias for temporal reasoning and knowledge sharing across time series.

3 METHODOLOGY

TEMPO combines local seasonal-trend decomposition, adaptive prompting, and a GPT backbone to represent and forecast multivariate time series. Decomposed component representations are processed separately and recombined into the final forecast.

  • Problem definition: Multivariate forecasting predicts the next H timestamps from the previous K observations for each feature channel.
  • Problem definition: TEMPO uses zero-shot learning in which the model does not access the target dataset’s history or horizon values during training.
  • Time-series input representation: STL decomposes each input into trend, seasonal, and residual components to represent long-term patterns, repeating cycles, and remaining variation.
  • Time-series input representation: Local decomposition uses learnable parameters and an MSE reconstruction loss to align component estimates with global STL decomposition.
  • Time-series input representation: When trend and seasonal components are non-orthogonal, orthogonal bases cannot fully separate them, motivating explicit decomposition before attention.
  • Prompt design: TEMPO aggregates patched temporal embeddings and concatenates adaptive prompts with trend, seasonality, and residual representations before GPT processing.
  • Prompt design: The prompt pool is designed to accommodate non-stationarity and distribution shifts across diverse time-series datasets.
  • Generative pretrained transformer architecture: GPT outputs are split by temporal component, transformed into component forecasts, and additively recombined into the final prediction.

4 EXPERIMENTS

TEMPO is evaluated in zero-shot forecasting across diverse benchmark, financial, and news datasets, including settings with contextual information. It outperforms baseline methods across long-term horizons and cross-domain sectors.

  • Datasets and settings: TEMPO is evaluated on diverse time series datasets spanning electricity, traffic, climate, news, and finance, with frequencies from minutes to quarters.The experiments include benchmark datasets, GDELT, and the proposed TETS dataset.
  • Zero-shot long-term forecasting: Under the many-to-one setting, TEMPO achieves the highest average performance across datasets and prediction horizons.Table 1 reports MSE and MAE for prediction lengths O ∈{96, 192, 336, 720}, with lower errors indicating better forecasts.
  • Zero-shot long-term forecasting: TEMPO improves MAE over PatchTST by around 6.5% on Weather and 19.1% on ETTm1.These improvements are reported relative to the previous state-of-the-art model under the many-to-one transfer-learning setting.
  • Contextual-information forecasting: The contextual-information experiments use TETS, a benchmark combining S&P 500 time series with contextual information, and GDELT, evaluated with SMAPE.TETS contains financial-sector data, while GDELT provides event-related information.
  • Contextual-information forecasting: In cross-domain sectors, TEMPO combines time series with contextual summaries and outperforms all baseline methods on TETS and GDELT.The many-to-many setting trains on in-domain sectors and directly tests on unseen cross-domain sectors using SMAPE.

5 ANALYSIS

The analysis examines how prompts and decomposition affect TEMPO and how decomposed components contribute to predictions. Both components are presented as important for robust zero-shot forecasting, with seasonality dominant in ETTm1.

  • Ablation study: Removing the prompt component deteriorates predictive accuracy on average across the evaluated datasets.The ablation compares TEMPO with variants without prompts, decomposition, or decomposition-loss alignment using MSE and MAE.
  • Ablation study: Decomposition loss improves the quality of individual trend, seasonal, and residual components, which is important for forecasting accuracy.The analysis links decomposition-loss alignment to use of a richer historical dataset.
  • Ablation study: Prompting without decomposition can harm the backbone model’s performance in most cases.The paper attributes this to difficulty prompting raw time series with limited semantic information.
  • Component analysis: SHAP analysis for ETTm1 indicates that the seasonal component has the dominant influence on TEMPO’s predictions.The analysis interprets this as strong model dependence on recurring patterns in the data.

6 CONCLUSION

TEMPO combines soft prompts and seasonal-trend decomposition in a pretrained Transformer for zero-shot forecasting. Experiments report strong accuracy and generalizability across datasets, long-term horizons, and multimodal inputs.

  • TEMPO integrates soft prompts and seasonal-trend decomposition within a pretrained Transformer backbone for zero-shot time series forecasting.The design targets different temporal-semantic components and supports multimodal inputs.
  • TEMPO predictions closely track ground truth across high-variability, periodic, and changing-trend series at prediction lengths 96 and 192.The visual comparisons include ETTh1, ETTh2, ETTm1, and ETTm2 examples against GPT4TS.
  • TEMPO is compared with GPT4TS on long-term forecasting examples from ETTh1, ETTh2, ETTm1, ETTm2, and Weather datasets.The cited figures visualize predicted values under different temporal characteristics and forecasting lengths.
  • TEMPO is also evaluated against TimeGPT on long-term forecasting examples from ETTm1 and ETTm2.The comparison extends the evaluation beyond GPT4TS to another foundation-model forecaster.
  • TEMPO outperforms TimeGPT in the reported zero-shot long-term forecasting comparison, while the paper notes that TimeGPT may perform less well in such scenarios.The authors frame this as evidence for models adapting across application breadth and forecasting-horizon depth.

B.1 TOWARDS FOUNDATION MODEL EXPERIMENTS DETAILS

The foundation-model experiments use standard benchmark datasets and a cross-domain zero-shot setup. The study also reports repeated iterative evaluation and hyperparameter exploration for GPT layers and reconstruction loss.

  • The benchmark suite includes ETTm1, ETTm2, ETTh1, ETTh2, Weather, Electricity, Traffic, ILI, and exchange datasets.The datasets cover electricity load, weather, and other time series domains.
  • Three iterative loops are used to average outcomes while exploring 3–6 GPT layers and reconstruction-loss weights of 0.001, 0.01, and 1.The documented settings report optimal search results, while a comprehensive layer-count analysis is deferred to future work.
  • For each prediction length, models train on pooled data from multiple domains and test on an unseen target domain.Training examples are fully shuffled and balanced by selecting equal numbers from each domain.

B.2 PROPOSED TETS DATASET SETTING

The TETS setting forecasts quarterly S&P 500 company EBITDA from financial statements and contextual news, including sector-based zero-shot evaluation. It uses channel-independent forecasting, rolling windows, and SMAPE-based assessment.

  • Data Collection: TETS uses financial statements from the 500 largest U.S. companies across 11 S&P 500 sectors and adds condensed news context collected through the ChatGPT API.Seven sectors support training and evaluation, while four sectors are reserved for zero-shot forecasting.
  • Prediction objective: The prediction target is quarterly EBITDA from 2000 to 2022, using foundational income-statement metrics as input features.The experiment forecasts company earnings-related financial time series.
  • Prediction objective: TEMPO, GPT4TS, and T5 use channel independence and univariate forecasting, whereas other models use multivariate inputs to predict EBITDA.All five features are used for training in the language-based models, but only EBITDA is available during training for the target series.
  • Prediction objective: The model uses the past 20 quarters to forecast the next quarter iteratively four times, producing a yearly prediction evaluated with SMAPE.The cumulative yearly value is the sum of four quarterly predictions, and SMAPE is also used as the forecasting loss.
  • Data Split: The split produces 80,600 in-domain and 9,199 zero-shot samples, using 70% of in-domain data for training, 10% for evaluation, and all zero-shot data for testing.The channel-independent setting yields five times as many samples as the channel-dependent setting.
  • Evaluation: SMAPE-Abs is used because financial metrics vary substantially across companies and EBITDA can include negative values.SMAPE may reach 200% when true and predicted values have opposite signs, so outliers at the 80% and 90% thresholds are filtered.
  • Additional setting: The GDELT experiment predicts NumMentions, NumArticles, and NumSources for event types across 55 U.S. regions and national U.S. data.The event root types are divided into seen and unseen sets for zero-shot evaluation.

C FURTHER RESULTS

Further experiments examine self-supervised cross-domain transfer, comparisons with full-shot specialized models and ARIMA, and prompt-pool design. Results support TEMPO’s transferability and show that semantic prompt designs and prompt pools can improve adaptation.

  • C.1 SELF-SUPERVISED REPRESENTATION LEARNING: TEMPO is pretrained across domains, fine-tuned with 5% of downstream training data, and achieves the lowest MSE and MAE in most unseen-domain cases.On Weather, its average MSE and MAE are 7.3% and 4.6% lower than GPT2, respectively.
  • C.2 COMPARING WITH FULL-SHOT STATE-OF-THE-ARTS BASELINES: TEMPO in zero-shot mode surpasses state-of-the-art full-shot models on ETTh1 and ETTh2.The comparison includes iTransformer, Crossformer, TiDE, and SCINet.
  • C.3 COMPARING WITH ARIMA: TEMPO retains predictive accuracy beyond its training scope, unlike ARIMA’s need for recalibration across disparate time series.The comparison is reported in Table 8.
  • D.1 DESIGN OF PROMPT POOL: A shared prompt pool retrieves top-K prompts from patched time-series inputs so similar series can receive similar temporal knowledge.The prompts can encode dependencies, trends, and seasonality, and selected values are concatenated with trend, seasonal, and residual embeddings.
  • D.2 RESULTS ON DIFFERENT PROMPT DESIGN: In ETTm2, the prompt pool outperforms the semi-soft prompt in three of four scenarios.Semi-soft and hard prompts also outperform simple soft prompts, indicating a benefit from explicit semantic content.
  • D.1 DESIGN OF PROMPT POOL: Prompt-pool experiments tune pool size, top-k count, and prompt length across 27 combinations, with long- and short-term settings using M = 30 and K = 3.The prompt values and model parameters are trained end-to-end, while pool prompts are initially random.
  • D.3.1 PROMPT SELECTION DISTRIBUTION: Prompt-selection histograms show different preferences for trend, seasonal, and residual components, with three prompts selected per component.In ETTm2, prompts 11, 20, and 24 primarily capture trends, while 8, 10, and 29 primarily capture seasonal fluctuations.

D.4 ANALYSIS ON DESIGNS OF INJECTING CONTEXTUAL INFORMATION

The section examines prompt designs for injecting textual context and compares TEMPO with other models’ representations and training time. The supplied passages describe the evaluation setup and interpretive visualizations, but do not report numerical SMAPE values or specific runtime outcomes.

  • Contextual prompt designs: Table 10 reports SMAPE results for the GEBDIT dataset under different textual-information injection designs.
  • Contextual prompt designs: Eight distinct prompt designs were formulated to investigate textual injection and original prompt design from a multi-modality perspective.“Sum” uses a direct textual summary, “SumP” uses textual information as a query within the prompt pool, and “+” and “⊕” denote summation and concatenation.
  • Representation analysis: TEMPO’s decomposed-component representation exhibits soft boundaries between trend, seasonality, and residual components, whereas GPT4TS representations are centered more collectively.
  • Representation analysis: Figure 12 compares GPT4TS and TEMPO representations for prediction length O = 96 using t-SNE, distinguishing trend, seasonality, and residual components.
  • Training-time comparison: Figure 13 presents each model’s training time as a ratio relative to TEMPO under a channel-independent setting.Ratios below 1 indicate faster training than TEMPO, while ratios above 1 indicate slower training.

E THE ROLES OF GENERALIZED ADDITIVE MODELS (GAM) AND SHAPLEY ADDITIVE EXPLANATIONS (SHAP)

GAM and SHAP are used to interpret TEMPO by modeling additive feature effects and attributing complex-model predictions. The analysis relates component contributions to dataset seasonality and supports more transparent, targeted interpretation.

  • Interpretability tools: GAM provides intrinsic interpretability by modeling feature effects as additive components in TEMPO.It is also used to examine how individual features contribute to the final prediction.
  • Interpretability tools: SHAP attributes feature effects post hoc to explain predictions from otherwise opaque complex models.
  • Interpretability tools: The analyses quantitatively test assumptions about model behavior and can reveal unexpected component effects, including unusually high residual impact associated with overfitting noise.
  • Dataset analysis: SHAP analysis compares component contributions using ETTm1 and weather examples, with full values reported in Table 11.
  • Dataset analysis: ETTm1 has seasonality strength 0.99, while weather has strength 0.476 and a more pronounced trend.
  • Dataset analysis: For ETTm1, increasing prediction length degrades performance primarily because seasonal-term predictions become inaccurate.

G.1 PROOF OF THEOREM 3.1

The theoretical analysis shows that non-orthogonal trend and seasonal signals cannot be separated into disjoint orthogonal bases, limiting automatic disentanglement by PCA-like self-attention. A complementary frequency-domain argument explains how STL simplifies forecasting by separating spectral components.

  • Theorem 3.1: If seasonal S(t) and trend T(t) signals are non-orthogonal, their active orthogonal-basis subsets must overlap.
  • Theorem 3.1: The proof expands S(t) and T(t) in orthogonal bases and uses their nonzero inner product to show that at least one basis vector contributes to both signals.
  • Theorem 3.1: Because PCA learns orthogonal bases and self-attention in pretrained large models is closely related to PCA, automatic trend-seasonality separation requires manual decomposition.
  • Frequency-domain analysis: Forecasting can be viewed in the frequency domain as predicting the evolution of the frequencies composing a time-series signal.
  • Frequency-domain analysis: Under Proposition G.2, predicting the next frequency-domain value determines the next time-domain sample and the remaining updated frequency values.
  • Frequency-domain analysis: STL decomposition simplifies forecasting by separating components with distinct dominant periodic patterns and reducing intertwined periodic influences.

H DETAIL OF THE TETS DATASET

The TETS dataset combines company financial time series with contextual information and supports both in-domain and zero-shot sector splits. Its construction includes interpolation and zero-filling choices for missing financial values, alongside generated textual context examples.

  • Financial data: The dataset uses company balance sheets, income statements, and cash-flow statements as primary financial sources.
  • Dataset composition: It includes 11 S&P 500 sectors, with sector-specific company counts ranging from 21 in Basic Materials to 73 in Industrials.
  • Dataset composition: The sector division supports both in-domain and zero-shot task settings.
  • Data preparation: Missing pre-2010 numerical data are linearly interpolated, while missing research-and-development expenses are filled with zero because null entries typically indicate no investment.
  • Contextual information: Financial time series primarily capture internal historical information, whereas external market information also affects companies’ future development.
  • Contextual information: ChatGPT API is used to filter and summarize relevant news and reports into contextual information, with prompts combining company, period, query, and requirements.Responses are capped at 110 tokens in the described procedure.
Loading 2310.04948v3…