PaperlayerView exact revision on arXiv

Source-linked AI summary

Compositional Spectral Prompts for LLM-based Online Time Series Forecasting

Seungyoon Choi, Hyunchul Kim, Jae-Gil Lee, Chanyoung Park

arXiv:2609.02093v1cs.LG

TL;DR

Memory-buffer-based online forecasting methods struggle with long-term adaptation and unseen patterns. CoSPOT uses compositional spectral prompts to guide a frozen LLM forecaster as data evolve. Experiments report that it consistently outperforms state-of-the-art online forecasting methods with extremely few online parameter updates.

  • Problem

    Existing memory buffer-based online time series forecasting methods struggle with long-term adaptation and generalization to unseen patterns.

    Source: §1 · p. 2

  • Method

    CoSPOT guides a frozen pre-trained LLM forecaster with compositional prompts built from frequency-domain spectral bases, tuning only the output projection layer online.

    Source: §1 · p. 2, §4.3 · p. 5

  • Results

    CoSPOT consistently outperforms state-of-the-art online time series forecasting methods with extremely few online parameter updates.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Unseen patterns can be represented by recombining a finite set of learned spectral basis prompts without additional training.

    Source: §4.2 · p. 5

Abstract

from arXiv · show

To address the sequential and evolving nature of time series, the Online Time Series Forecasting (OTSF) task has been extensively studied in multiple domains. Existing research focuses on adapting to non-stationary environments by employing memory buffer-based retrieval strategies. However, we observe that such frameworks struggle with long-term adaptation and fail to generalize to unseen patterns. To this end, we introduce CoSPOT, an LLM-based online time series forecasting framework that leverages a pre-trained LLM as the backbone online forecaster, motivated by its strong few-shot capabilities. For efficient online adaptation, CoSPOT keeps the LLM frozen and employs compositional spectral prompts grounded in frequency-domain bases to guide the model with the overall distribution of the input, thereby substantially reducing the number of parameters updated during the online phase. Specifically, CoSPOT decomposes time series into frequency bases and composes the corresponding spectral basis prompts according to their amplitudes, allowing unseen patterns to be represented as new combinations of learned basis prompts. Our extensive experiments on real-world datasets demonstrate the superiority and practicality of CoSPOT across challenging online scenarios, including extended online phases and cross-dataset settings with substantial distribution shifts. Our code is available at https://github.com/seungyoon-Choi/CoSPOT.

1 Introduction

CoSPOT addresses the long-term adaptation and unseen-pattern weaknesses of memory-buffer-based online time series forecasting with an LLM guided by compositional spectral prompts. Its introduction reports that CoSPOT consistently outperforms state-of-the-art OTSF methods across online scenarios with extremely few online parameter updates.

Motivation

  • Online learning enables incremental adaptation to evolving time series, whereas static batch methods can fail under change and frequent retraining is impractical.Prior LLM forecasting work [8] [28] motivates using pretrained models for few-shot transfer under limited online data.Source: §1 · pp. 1–2

Challenges in OTSF

  • Memory-buffer-based methods including FSNet [13], OneNet, and DSOF struggle with extended online phases and unseen patterns because retrieval capacity is limited and novel patterns are unreliable matches.Longer online phases increase the diversity of recurring patterns and can cause catastrophic forgetting; distribution shifts can make retrieval from previously learned patterns fail.Source: §1 · pp. 1–2

Motivation for CoSPOT

  • LLM-based forecasters TimeLLM [8], LLM4TS, and GPT4TS [28] perform comparably to DSOF [10] in extended online phases but struggle to generalize to pattern shifts.The cited comparison follows the models’ original training protocols and fine-tunes only the output projection online.Source: §1 · p. 2

Compositional spectral prompting

  • CoSPOT decomposes inputs with the DFT and combines learnable frequency-basis prompts according to component amplitudes, guiding a frozen LLM with input-specific prompts.The prompts represent unseen patterns as new combinations of learned spectral bases, while the prompt bank and LLM remain frozen during online adaptation.Source: §1 · p. 2

Contributions

  • CoSPOT consistently outperforms state-of-the-art OTSF methods across varied online scenarios with extremely few online parameter updates.The introduction presents this as a finding from extensive experiments.Source: §1 · p. 2

2 Related Works

The related work spans online forecasting, LLM-based forecasting, prompt-based continual learning, and frequency analysis. CoSPOT combines these directions by using frequency-grounded prompts for online adaptation to changing time-series distributions.

Online Time Series Forecasting

  • FSNet [13], OneNet, and DSOF advance online forecasting, but prior methods do not explicitly model input patterns and explicit pattern storage limits adaptation to unseen distributions and extended online phases.DSOF also redefines the OTSF setting to address information leakage and uses a dual-stream mechanism to update model parameters.Source: §2 · pp. 2–3

Time Series Forecasting with LLMs

  • LLM4TS, GPT4TS [28], and TimeLLM [8] adapt pretrained LLMs to time-series forecasting, motivating CoSPOT’s use of LLMs for rapid online adjustment.Their approaches include two-stage fine-tuning, retraining positional embeddings and normalization layers, and aligning time series with word embeddings, respectively.Source: §2 · p. 3

Prompt-based Continual Learning

  • Prompt-based continual learning methods such as VPT [7], L2P [17], and S-Prompts [16] tune prompts while keeping core model parameters unchanged, but time-series distribution shifts remain unexplored.CoSPOT is presented as the first efficient, scalable prompting method for online learning, encoding knowledge from underlying frequency bases; online learning better fits continuous time series than continual learning’s distinct-task assumption.Source: §2 · p. 3

Frequency Analysis in Time Series Forecasting

  • DFT and DWT capture global and localized multiscale frequency information, respectively, while CoSPOT composes frequency-grounded prompts to represent unobserved patterns under distribution shifts.FEDformer [27] uses DFT and DWT in Fourier- and Wavelet-Enhanced Structures, but frequency-domain methods mainly exploit coefficients rather than learning prompt-level knowledge for each frequency basis.Source: §2 · p. 3

3 Preliminaries

The preliminaries define forecasting from a length-L history to H-step predictions under MSE, then formalize OTSF as static training followed by sequential online updates evaluated by cumulative MSE.

3.1 Time Series Forecasting

  • The dataset is chronologically split into training, validation, and online timestamps; forecasting maps a length-L look-back window to H future steps and minimizes MSE.At each time i, the model predicts the following H steps from the preceding L observations.Source: §3.1 · p. 3

3.2 Online Time Series Forecasting

  • OTSF trains on (L+H)-sized sequences using static batch learning, then streams online timestamps with a size-1 moving window for real-time updates, evaluating cumulative MSE over H steps.Source: §3.2 · p. 3

4 Proposed Method: CoSPOT

CoSPOT combines a frozen pre-trained LLM, a time-series backbone, and recent textual context with compositional spectral prompts. Its online framework adapts to changing patterns while limiting parameter updates.

4.1 Enhancing model adaptability using a pre-trained LLM

  • For data-scarce adaptation, CoSPOT pairs a frozen pre-trained LLM and time-series backbone with text describing task, dataset, recent values, and recent frequencies.The text supplies short-term context complementary to the spectral prompt, without requiring additional training.Source: §4.1 · pp. 3–4

4.2 Robust Adaptation via Compositional Spectral Prompting

  • Compositional spectral prompts represent emerging patterns by recombining learned frequency-basis prompts weighted by input amplitudes, without updating the LLM online.A low-pass cutoff removes high-frequency bases to focus prompts on overall patterns and reduce noise.Source: §4.2 · pp. 4–5

4.3 Overall Framework

  • A pre-developed alignment module maps the time-series backbone representation toward the natural-language modality for use with the LLM.The paper adopts prior alignment modules rather than focusing on alignment design.Source: §4.3 · p. 5
  • During training, CoSPOT trains the backbone, alignment module, spectral prompt bank, and output projection; online, it tunes only the output projection.The LLM remains frozen throughout, and the other trained components also remain frozen online.Source: §4.3 · p. 5
  • To update at each step without using unavailable future targets, CoSPOT uses the next ground-truth value and pseudo-labels for the remaining forecast horizon.A frozen projection layer generates pseudo-labels from the backbone representation, and a geometric decay factor reduces the online loss impact of farther-horizon predictions.Source: §4.3 · pp. 5–6

5 Experiments

Experiments evaluate CoSPOT on five forecasting benchmarks against static, foundation, LLM-based, and online forecasting baselines. The setup combines a frozen pre-trained LLM with spectral prompts and online output-layer updates.

Datasets

  • The evaluation covers five benchmarks spanning electricity, weather, traffic, and exchange rates, using 20%/5%/75% training, validation, and test splits.Exchange Rate is highlighted as a robustness test because its weaker periodicity and exposure to random economic and market events differ from strongly seasonal datasets.Source: §5 · p. 6

Baselines

  • Baselines span static forecasting, time-series foundation models, LLM-based models, and online forecasting, including DLinear [23], Chronos-2 [1], GPT4TS [28], and FSNet [13].Other listed models include PatchTST [12], iTransformer [11], TimeMixer [15], TimesFM [4], LLM4TS, Time-LLM [8], OneNet, and DSOF.Source: §5 · pp. 6–7

Training and Online Phases

  • Training freezes the pre-trained LLM while learning other parameters; online, all parameters except the output projection layer are frozen and that layer is updated per instance.Each input is aligned and text-described, decomposed by DFT into frequency bases and amplitudes, and represented by composed spectral basis prompts alongside text and aligned embeddings for LLM projection.Source: §5 · p. 6

Implementation Details

  • Experiments use prediction lengths H=1, 24, and 48 with lookback L=96, PatchTST [12] as the backbone, and Llama-7B [14] as the default LLM.The protocol uses MSE and MAE, trains for 10 epochs with AdamW, and performs one-step online updates per data instance; specified hyperparameters are γ=0.3, j=2, and δ=0.8.Source: §5 · p. 7

5.1 Overall Performance

CoSPOT outperforms the reported online forecasting, foundation-model, and LLM-based baselines, with robust performance across multiple datasets. The results also show that several competing online methods struggle to adapt to evolving patterns, particularly when data are scarce or distributions shift continuously.

5.1 Overall Performance

  • CoSPOT outperforms Chronos-2, TimesFM, LLM4TS, GPT4TS, TimeLLM, and DSOF in the reported eight-dataset evaluation.The reported results are averages over three runs; CoSPOT's robustness is attributed to combining a pre-trained LLM with compositional spectral prompts.Source: §5.1 · p. 7
  • Without information leakage, FSNet and OneNet underperform static models DLinear, iTransformer, and TimeMixer.The authors attribute this to FSNet and OneNet exploiting leaked training data rather than learning newly arriving patterns or adapting to them.Source: §5.1 · p. 7
  • DSOF adapts quickly without leakage, but its focus on rapid convergence leaves it poorly equipped to adapt when data are scarce.CoSPOT combines spectral prompts and text descriptions to guide the model with underlying patterns and recent context, supporting adaptation under continuous distribution shifts.Source: §5.1 · p. 7

5.2 Ablation Study

The ablation study finds that compositional spectral prompts and text descriptions each help online forecasting, with their combination yielding synergistic effects. It also finds DWT more effective than STFT for recent-pattern information on challenging, longer-horizon tasks.

5.2 Ablation Study

  • Compositional spectral prompting yields nearly a 50% improvement in MSE by recombining learned frequency-basis prompts to adapt to unseen patterns.The study compares Row (1) with Row (2).Source: §5.2 · p. 7
  • Text descriptions improve forecasting by injecting dynamically changing recent-pattern context beyond raw numerical observations, without additional training.The study compares Row (1) with Row (3), and describes the text modality as useful in data-scarce online scenarios.Source: §5.2 · p. 7
  • Combining spectral prompts with text descriptions yields synergistic effects: overall patterns and recent time- and frequency-domain patterns jointly support adaptation despite distribution shifts.The study compares Rows (2&3) with vanilla CoSPOT, Row (4)-1.Source: §5.2 · p. 7
  • DWT is more effective than STFT for recent-pattern information on longer-horizon tasks because its frequency-dependent window size better captures non-stationary signals.The methods perform comparably on easier prediction-horizon-1 tasks; the study attributes STFT's limitation to its fixed window size.Source: §5.2 · p. 7

5.3 Further Analysis

Further analyses find that CoSPOT remains adaptable during extended online phases and distribution shifts, while using few online parameter updates and compositional spectral prompts to represent emerging patterns.

5.3.1 Robustness to the extension of the online phase.

  • CoSPOT maintains adaptability comparable to Table 1 when the online phase is extended, while prior methods degrade as recurring and unseen patterns accumulate.The evaluation changes the train/valid/test split from 20%/5%/75% to 10%/5%/85%; CoSPOT’s compositional spectral prompts and recent-pattern text help adapt in data-scarce scenarios.Source: §5.3.1 · pp. 7–8, Table 3 · p. 8

5.3.2 Robustness to distribution shifts.

  • CoSPOT outperforms all baselines across the 8 cross-dataset scenarios, where different ETT datasets induce distribution shifts between training and online phases.Unlike associative-memory methods that fail when unseen patterns arise, CoSPOT composes spectral basis prompts to represent them and maintains performance largely consistent with Table 1.Source: §5.3.2 · pp. 8–9, Table 4 · p. 9

5.3.3 Online Updating Cost.

  • CoSPOT updates significantly fewer parameters than baselines, with only slightly longer online-phase duration and inference latency but higher initial training cost.The analysis prioritizes online cost because OTSF continually adapts to a data stream after initial training; CoSPOT’s online-phase cost is described as comparable despite its larger training overhead.Source: Table 5 · p. 9, §5.3.3 · p. 9, Table 6 · p. 9

5.3.4 Generalizability of Compositional Spectral Prompting.

  • With its prompt bank frozen online, CoSPOT’s compositional spectral prompting can represent incoming unseen patterns by recombining knowledge learned from frequency bases.The comparison considers an optimal bank further trained online, a CoSPOT bank frozen online, and a randomly generated bank.Source: §5.3.4 · p. 9

5.3.5 Comparison of frequency- and time-domain prompting.

  • Under distribution shifts, frozen frequency-domain prompting remains robust, whereas frozen time-domain prompting substantially degrades; updating prompts online improves performance in both domains.Frequency-domain prompts compose decomposed frequency components, while the time-domain alternative retrieves prompts using representation-space prototypes; finalized frequency prompts also show higher similarity across online-trained and frozen settings.Source: Table 8 · p. 10, §5.3.5 · p. 10, Table 9 · p. 10

5.3.6 Sensitivity Analysis.

  • On ETTh2 and ETTm1, γ=0.3 performs best, while δ around 0.8 performs best for horizons 24 and 48; retaining high-frequency components or eliminating decay hurts MSE.The authors attribute high-frequency components to noise in overall-distribution prompts, and use geometric decay to mitigate unreliable pseudo-labels and less accurate predictions at distant timestamps.Source: Figure 3 · p. 10, §5.3.6 · pp. 10–11, Figure 4 · p. 10

6 Conclusion

CoSPOT is an LLM-based online time series forecasting framework that combines distribution-aware spectral prompts with recent-pattern text descriptions to support adaptation under shifting distributions and extended online phases.

6 Conclusion

  • CoSPOT recombines learnable spectral basis prompts according to each input’s frequency structure to capture its overall distribution and guide a pre-trained LLM’s rapid adaptation.The approach targets continuously shifting distributions and data-scarce scenarios.Source: §6 · p. 11
  • Recent-pattern text descriptions enrich limited online data without requiring additional training.Source: §6 · p. 11
  • CoSPOT shows promising performance across real-world datasets and robustness to distribution shifts and extended online phases.The conclusion highlights its applicability to real-world online forecasting.Source: §6 · p. 11

GenAI Disclosure

The authors report limited LLM assistance for language editing and minor code refactoring/debugging, while stating that the paper’s core contributions are original.

GenAI Disclosure

  • LLMs were used for language editing and minor plotting/visualization code refactoring and debugging; the authors reviewed the changes and say the core work is original.The authors identify the core ideas, methodology, experiments, and interpretations as entirely their original contributions.Source: GenAI Disclosure · p. 11

Related papers

Loading 2609.02093v1…