PaperlayerView exact revision on arXiv

Source-linked AI summary

Context-Alignment: Activating and Enhancing LLM Capabilities in Time Series

Yuxiao Hu, Qian Li, Dongxiao Zhang, Jinyue Yan, Yuntian Chen

arXiv:2501.03747v3cs.LGcs.CLstat.AP

TL;DR

LLM-based time-series methods must both activate and enhance language-model capabilities, but token-level alignment overlooks LLMs’ deeper understanding of linguistic logic and structure. The paper proposes Context-Alignment through DSCA-GNNs and FSCA to build coherent TS-language context, and reports effectiveness across tasks, particularly in few-shot and zero-shot forecasting.

  • Problem

    Token-level alignment between TS and language embeddings may not fully activate LLMs because it overlooks their strength in understanding linguistic logic and structure.

  • Method

    Context-Alignment uses DSCA-GNNs for structural alignment with dual-scale nodes and logical alignment with directed edges, instantiated with few-shot prompting as FSCA.

  • Results

    Experiments across multiple datasets and TS tasks show that FSCA surpasses existing techniques, especially in few-shot and zero-shot forecasting, while ablations emphasize Context-Alignment’s importance.

  • Takeaways & Limitations

    Context-Alignment provides a context-level approach for activating LLM capabilities on TS tasks by preserving structure and guiding logical relationships between TS data and language.

Abstract

from arXiv · show

Recently, leveraging pre-trained Large Language Models (LLMs) for time series (TS) tasks has gained increasing attention, which involves activating and enhancing LLMs' capabilities. Many methods aim to activate LLMs' capabilities based on token-level alignment, but overlook LLMs' inherent strength in natural language processing -- \textit{their deep understanding of linguistic logic and structure rather than superficial embedding processing.} We propose Context-Alignment (CA), a new paradigm that aligns TS with a linguistic component in the language environments familiar to LLMs to enable LLMs to contextualize and comprehend TS data, thereby activating their capabilities. Specifically, such context-level alignment comprises structural alignment and logical alignment, which is achieved by Dual-Scale Context-Alignment GNNs (DSCA-GNNs) applied to TS-language multimodal inputs. Structural alignment utilizes dual-scale nodes to describe hierarchical structure in TS-language, enabling LLMs to treat long TS data as a whole linguistic component while preserving intrinsic token features. Logical alignment uses directed edges to guide logical relationships, ensuring coherence in the contextual semantics. Following the DSCA-GNNs framework, we propose an instantiation method of CA, termed Few-Shot prompting Context-Alignment (FSCA), to enhance the capabilities of pre-trained LLMs in handling TS tasks. FSCA can be flexibly and repeatedly integrated into various layers of pre-trained LLMs to improve awareness of logic and structure, thereby enhancing performance. Extensive experiments show the effectiveness of FSCA and the importance of Context-Alignment across tasks, particularly in few-shot and zero-shot forecasting, confirming that Context-Alignment provides powerful prior knowledge on context. The code is open-sourced at https://github.com/tokaka22/ICLR25-FSCA.

1 Introduction

The paper argues that token-level alignment does not fully activate LLMs for time-series tasks because it overlooks their strengths in linguistic logic and structure. It proposes Context-Alignment and the DSCA-GNNs/FSCA framework to construct coherent TS-language context and improve performance, especially in few-shot and zero-shot forecasting.

  • Motivation: LLM-based TS methods face a mismatch between language-model training data and time-series data, creating challenges for understanding TS inputs and improving task performance.The paper frames these as two sequential issues: activating LLM capabilities for TS and enhancing their TS-task performance.
  • Motivation: Token-level alignment may underuse LLMs because their strength lies in understanding linguistic logic and structure rather than superficial token processing.The paper contrasts token-embedding alignment with context-level comprehension.
  • Contribution: Context-Alignment aligns TS data with a linguistic component in LLMs’ familiar language environment to construct consistent TS-language context.The paradigm is intended to activate LLM capabilities by enabling contextualization and comprehension of TS data.
  • Contribution: DSCA-GNNs achieve structural alignment with dual-scale nodes and logical alignment with directed edges, while FSCA integrates few-shot prompting to enhance TS-task performance.The framework can be integrated into various layers of pre-trained LLMs and targets awareness of logic and structure.
  • Results: Experiments across multiple datasets and TS tasks surpass existing techniques, particularly for few-shot and zero-shot forecasting, while ablations emphasize Context-Alignment’s importance.The reported findings support both the effectiveness of the method and the role of Context-Alignment.

2 Related Work

Related work spans statistical and neural approaches for time-series analysis, while LLM-based methods increasingly target TS tasks. However, some methods enhance LLM performance without first enabling LLMs to understand TS inputs, and token-embedding alignment can overlook contextual coherence.

  • Time-series methods: Time-series research has progressed from statistical methods such as ARIMA and Prophet to CNN-, RNN-, and Transformer-based approaches.These methods model trends, seasonality, automatically extracted features, sequence prediction, and complex dependencies.
  • LLM-based methods: Recent LLM-based TS methods enhance capabilities through distribution adaptation or statistical prompting, but some overlook the need to first understand TS inputs.The related work distinguishes direct performance enhancement from input-understanding approaches.
  • Alignment limitation: Token-embedding alignment without contextual coherence and consistency fails to leverage LLMs’ inherent advantage in language understanding.The limitation concerns the relationship between aligned embeddings and coherent context.

3 Methodology

The methodology aligns time-series and language inputs at the context level by modeling their structure and logical relationships with DSCA-GNNs. VCA provides a vanilla-prompt instantiation, while FSCA extends the framework with few-shot demonstrations and integrates it into pre-trained LLMs.

  • Context-Alignment: DSCA-GNNs achieve Context-Alignment through dual-scale nodes for structural alignment and directed edges for logical alignment.The graph structure depends on the language prompts used, so different prompting methods yield different DSCA-GNN frameworks.
  • VCA: Vanilla prompting concatenates TS and language embeddings, but the resulting verbose input lacks clear structural divisions and contextual coherence.This makes the TS embeddings difficult for LLMs to analyze and leaves the input without logical guidance.
  • Structural alignment: Structural alignment aggregates tokens from each modality into coarse-grained linguistic components while fine-grained nodes preserve token-level information.The coarse-grained branch represents TS and prompt components, while the fine-grained branch retains detailed embeddings for downstream TS tasks.
  • Logical alignment: Directed edges encode that preceding TS parts provide information to prompts and that prompts lead to the subsequent TS output.In the coarse-grained graph, TS-to-prompt edges receive TS information from preceding parts, while prompt-to-TS edges indicate the correct prediction target.
  • Dual-scale interaction: Learnable interaction transfers macroscopic information from the coarse-grained graph to the fine-grained graph while retaining detailed TS information.The transfer uses an assignment matrix and learnable weight matrix before updating the fine-grained node embeddings.
  • Integration: The aligned node embeddings are input to pre-trained LLMs, and DSCA-GNNs can be repeatedly integrated into various LLM layers.The fine-grained branch produces the output used for MSE loss against ground truth; VCA is the vanilla-prompt instantiation.
  • FSCA: FSCA divides ordered TS embeddings into N parts and constructs N −1 prediction demonstrations using preceding parts as inputs and subsequent parts as ground truth.The resulting TS-language sequence repeats the language prompt between successive TS parts before graph construction.

4 Experiments

Experiments evaluate Context-Alignment across forecasting and classification tasks, with FSCA generally outperforming baselines and ablations supporting structural and logical alignment.

  • Experimental Overview: FSCA surpasses baselines across long-term, short-term, few-shot, zero-shot forecasting, and classification experiments.The evaluation covers eight long-term forecasting datasets, M4 short-term forecasting, ETT few-shot and cross-domain zero-shot forecasting, and UEA classification.
  • Long-term Forecasting: 3.1% lower average MSE than PatchTST and 7.3%, 12.2%, and 16.6% lower than S2IP-LLM, Time-LLM, and GPT4TS, respectively, are reported for long-term forecasting.FSCA is evaluated across eight datasets and multiple prediction horizons using MSE and MAE.
  • Short-term Forecasting: FSCA remains competitive on M4 short-term forecasting, with robustness attributed to structural and logical alignment across varying sequence lengths.M4 evaluation uses SMAPE, MASE, and OWA across prediction horizons from 6 to 48.
  • Few-shot Forecasting: With 5% of training data, FSCA reduces average MSE by 6.7% versus S2IP-LLM, 9.4% versus Time-LLM, 10.8% versus GPT4TS, and 15.8% versus PatchTST.The paper attributes these gains to structural division and logic guidance supplying prior knowledge under limited data.
  • Zero-shot Forecasting: FSCA improves performance by 13.3% over PatchTST and by 18.3%, 17.7%, and 24.3% over S2IP-LLM, Time-LLM, and GPT4TS in zero-shot forecasting.The cross-domain setup trains on Dataset A and tests on Dataset B without using training data from Dataset B.
  • Classification: FSCA reaches 76.4% accuracy on UEA classification, 2.4% above the next-best model, extending the reported effectiveness beyond forecasting.Figure 2 averages accuracy across 10 UEA subsets.
  • Ablation Study: Removing dual-scale GNNs, randomizing adjacency, omitting the coarse-grained branch, or excluding input-stage GNNs reduces performance.The ablations associate these declines with incorrect logical information, weaker macro-structure understanding, verbose inputs, or reduced initial context alignment.

5 Conclusion

The paper proposes Context-Alignment to align time series with language context through structural and logical alignment, then instantiates it as FSCA for pre-trained LLMs. Experiments report stronger performance, especially in few-shot and zero-shot forecasting, while ablations support the importance of Context-Alignment.

  • Conclusion: Context-Alignment constructs context-level alignment between time series and language, using DSCA-GNNs for structural and logical alignment.FSCA adds demonstration-example prompting to enhance pre-trained LLM performance on time-series tasks.
  • Conclusion: Experiments report that FSCA significantly outperforms other methods, particularly in few-shot and zero-shot forecasting.Ablation studies further confirm the importance of Context-Alignment.

A.2 DATASET DETAILS

The evaluation covers long-term, short-term, and classification time-series tasks using established forecasting and UEA datasets. The section also describes prompt-specific DSCA-GNN construction for classification inputs.

  • Dataset details: Long-term forecasting uses eight multivariate datasets, including ETT, Illness, Weather, Electricity, and Traffic.The ETT datasets include ETTh1, ETTh2, ETTm1, and ETTm2 at varying resolutions.
  • Dataset details: Short-term forecasting uses the M4 benchmark, comprising 10,000 time series across six groups with annual-to-hourly sampling rates.
  • Dataset details: Time-series classification uses 10 multivariate UEA datasets, with tables reporting classes, series lengths, feature dimensions, and train/test sample sizes.
  • Evaluation metrics: Forecasting is evaluated with MSE and MAE, while short-term M4 forecasting uses SMAPE, MASE, and OWA; classification uses accuracy.
  • Classification prompting: For classification, FSCA supplies one example per category for binary datasets, whereas VCA is used for multi-category tasks because prompt length limits demonstrations.
  • Classification prompting: The classification graph treats fine-grained elements as nodes and uses directed edges to encode TS-to-prompt and prompt-to-label relationships.

C.1 Long-term Forecasting full results

FSCA shows strong forecasting performance across long-term and short-term settings, outperforming most baseline models. In long-term forecasting, it achieves a 3.1% average MSE reduction over PatchTST.

  • Long-term forecasting: 3.1% average MSE reduction over PatchTST is achieved by FSCA in long-term forecasting.FSCA also outperforms S2IP-LLM, Time-LLM, and GPT4TS by 7.3%, 12.2%, and 16.6%, respectively.
  • Short-term forecasting: FSCA consistently outperforms most baseline models under different frequency settings in short-term forecasting.

C.3 Few-shot Forecasting full results

FSCA improves few-shot forecasting across training-data regimes and prediction lengths. It outperforms the second-best method by 6.7% with 5% training data and reduces average MSE by 7.8% with 10%.

  • 5% training data: 6.7% improvement over S2IP-LLM is achieved by FSCA in few-shot forecasting with 5% training data.DLinear remains strong on the ETTm1 dataset, where it is the exception to FSCA’s broader enhancements.
  • 10% training data: 7.8% average MSE reduction over GPT4TS is achieved by FSCA in few-shot forecasting with 10% training data.

C.4 Zero-shot Forecasting full results

FSCA achieves optimal zero-shot forecasting performance across all reported settings. Its average improvement over PatchTST is 13.3%, with larger MSE reductions against the evaluated LLM-based methods.

  • Zero-shot forecasting: 13.3% average improvement over PatchTST is achieved by FSCA in zero-shot forecasting across prediction lengths.FSCA reduces MSE over S2IP-LLM, Time-LLM, and GPT4TS by 18.3%, 17.7%, and 24.3%, respectively.

C.5 Classification Full Results

FSCA achieves consistently superior performance across 10 multivariate UEA classification datasets, extending its demonstrated utility beyond forecasting. The classification results support the versatility of Context-Alignment across time-series tasks.

  • C.5 Classification Full Results: FSCA achieves consistently superior classification performance compared with classical, RNN-based, Transformer-based, MLP-based, and other LLM-based approaches.The evaluation covers 10 multivariate UEA classification datasets.
  • C.5 Classification Full Results: 2.4%, 2.7%, and 2.8% are FSCA’s average accuracy improvements over S2IP-LLM, Time-LLM, and GPT4TS, respectively.These comparisons quantify FSCA’s gains against three LLM-based baselines.
  • C.5 Classification Full Results: The classification results demonstrate that FSCA can be applied beyond time-series forecasting tasks.Table 15 reports the full classification results across the 10 UEA datasets.

F.2 Scale analysis

FSCA’s scaling behavior depends on model depth, training-data availability, and the number of few-shot examples. More training data improves performance, while deeper models and more examples can reduce effectiveness in some settings.

  • F.2 Scale analysis: Performance declines as the number of GPT-2 layers increases, consistent with findings reported for GPT4TS.This behavior is identified through ablation experiments on GPT-2 layer count.
  • F.2 Scale analysis: Using 5%, 10%, 25%, 50%, 75%, and full training data produces continuous performance improvement, with a particularly significant gain at 50%.The analysis uses results reported in Table 17.
  • F.2 Scale analysis: More few-shot examples yield modest gains for prediction length 96 but reduce effectiveness for prediction lengths 336 and 720.The authors attribute this pattern to shorter input lengths per example after dividing the time-series input, creating a mismatch with longer prediction horizons.
  • F.2 Scale analysis: Fully tuning FSCA and GPT4TS performs suboptimally, whereas FSCA focuses on freezing most LLM components to harness pretrained capabilities efficiently.The comparison is reported in Table 18.

F.5 Experimental efficiency analysis

FSCA is designed to improve time-series performance without the heavier prompt-retrieval operations used by several competing LLM-based methods. Its main added overhead comes from constructing coarse-grained inputs and using dual-scale GNN components, while graph-based alignment outperforms alternative network types.

  • F.5 Experimental efficiency analysis: FSCA ranks just behind GPT4TS in experimental efficiency among the compared LLM-based methods.The comparison considers parameter count and execution speed.
  • F.5 Experimental efficiency analysis: FSCA’s computational costs mainly arise from dual-scale GNN matrices and two linear layers that transform fine-grained node embeddings into coarse-grained inputs.The linear layers’ input dimension is determined by the number of input time-series patches.
  • F.5 Experimental efficiency analysis: The input-statistics prompt increases one-iteration training time from 0.587 seconds to 1.431 seconds because statistics must be recalculated and embeddings regenerated each iteration.The data-domain prompt performs nearly the same as the original prompt, while the input-statistics prompt slightly improves performance.
  • F.5 Experimental efficiency analysis: GCN-based Context-Alignment methods outperform MLP, CNN, and self-attention alternatives by modeling node-edge structural and logical relationships.Table 26 provides the network-type comparison.
  • F.5 Experimental efficiency analysis: FSCA outperforms the compared time-series foundation models on the reported classification and long-term forecasting tasks.The comparisons include UniTS-ST, MOMENT, and TSMixer.
Loading 2501.03747v3…