PaperlayerView exact revision on arXiv

Source-linked AI summary

TimeCMA: Towards LLM-Empowered Multivariate Time Series Forecasting via Cross-Modality Alignment

Chenxi Liu, Qianxiong Xu, Hao Miao, Sun Yang, Lingzheng Zhang, Cheng Long, Ziyue Li, Rui Zhao

arXiv:2406.01638v5cs.LGcs.AIcs.CL

TL;DR

Multivariate time series forecasting methods face limited learnable parameters, small training data, and entangled embeddings. TimeCMA aligns disentangled time-series embeddings with robust LLM prompt embeddings, outperforming all baselines across eight datasets while reducing computational costs through last-token processing.

  • Problem

    Existing multivariate time series forecasting methods have limited learnable parameters and small-scale training data, while LLM-based approaches struggle to learn disentangled embeddings.

  • Method

    TimeCMA uses dual-modality encoding and channel-wise cross-modality similarity retrieval to combine disentangled time-series embeddings with robust prompt embeddings.

  • Results

    TimeCMA outperforms all baselines in all cases, improving average MSE by 16.1% and MAE by 11.9% over OFA.

  • Takeaways & Limitations

    Using channel-wise alignment and only the stored last token offers an effective and computationally efficient approach to LLM-based multivariate forecasting.

Abstract

from arXiv · show

Multivariate time series forecasting (MTSF) aims to learn temporal dynamics among variables to forecast future time series. Existing statistical and deep learning-based methods suffer from limited learnable parameters and small-scale training data. Recently, large language models (LLMs) combining time series with textual prompts have achieved promising performance in MTSF. However, we discovered that current LLM-based solutions fall short in learning disentangled embeddings. We introduce TimeCMA, an intuitive yet effective framework for MTSF via cross-modality alignment. Specifically, we present a dual-modality encoding with two branches: the time series encoding branch extracts disentangled yet weak time series embeddings, and the LLM-empowered encoding branch wraps the same time series with text as prompts to obtain entangled yet robust prompt embeddings. As a result, such a cross-modality alignment retrieves both disentangled and robust time series embeddings, "the best of two worlds", from the prompt embeddings based on time series and prompt modality similarities. As another key design, to reduce the computational costs from time series with their length textual prompts, we design an effective prompt to encourage the most essential temporal information to be encapsulated in the last token: only the last token is passed to downstream prediction. We further store the last token embeddings to accelerate inference speed. Extensive experiments on eight real datasets demonstrate that TimeCMA outperforms state-of-the-arts.

Introduction

TimeCMA addresses weak and entangled embeddings in LLM-based multivariate time series forecasting through cross-modality alignment. It also reduces computational costs by retaining and storing only each prompt’s last-token embedding.

  • Motivation: MTSF methods are limited by few learnable parameters and small-scale training data, motivating the use of pre-trained LLMs for more robust embeddings.The introduction distinguishes statistical and deep learning-based methods before describing recent LLM integration.
  • Limitations of Existing Methods: Time series-based LLMs can produce weak embeddings from randomly initialized layers, while prompt-based approaches face data entanglement between time series and textual information.The cited passages identify the domain gap and noisy or entangled fused embeddings as central limitations.
  • TimeCMA Framework: TimeCMA uses dual-modality encoding and cross-modality alignment to retrieve disentangled, robust time series components from LLM-generated prompt embeddings via channel-wise similarity.The framework combines a time series encoding branch with an LLM-empowered encoding branch, then aligns their representations for forecasting.
  • Efficiency: Prompt-based LLM forecasting incurs high computational costs because multivariate series add variable and time dimensions and LLM outputs are computationally burdensome.The introduction contrasts multivariate inputs with one-dimensional prompt data and notes that freezing parameters does not remove the output burden.
  • Efficiency: TimeCMA instructs each variable’s prompt to place vital temporal information in its last token, feeds only that embedding for alignment, and stores it offline to accelerate inference.The last-token design preserves multivariate characteristics while reducing downstream computation through embedding reuse.

Related Work

Related work progresses from deep learning architectures for multivariate forecasting to LLM-based approaches. CNNs and Transformers capture temporal and variable dependencies, while LLMs leverage larger parameterization and training data but follow distinct prompting paradigms.

  • Deep Learning for Time Series Forecasting: CNNs capture variable and temporal correlations, whereas early Transformers often conflate variables observed at the same timestamp into one temporal token.This conflation can lead to suboptimal forecasting performance.
  • Deep Learning for Time Series Forecasting: iTransformer treats independent time series as tokens to better capture multivariate correlations, addressing a limitation of earlier Transformer-based methods.Despite such advances, deep learning methods remain constrained by limited parameterization and small-scale training data.
  • LLM-Based Time Series Forecasting: LLMs have achieved superior time series analysis performance by leveraging extensive parameterization and large-scale training data.LLM-based forecasting methods are categorized as time series-based or prompt-based according to whether prompts are included in the input.

Preliminaries

The preliminaries define multivariate time series and their prompt representation, then formalize forecasting future multivariate observations from historical data and prompts.

  • Multivariate Time Series: A multivariate time series is X = {x_1, ..., x_L} ∈ R^L×N, with L time steps and N variables.
  • Prompt: Each variable’s series is wrapped into a prompt containing words, time information, and values from L time steps.The prompts are P_S = {p_1, ..., p_N} ∈ R^S×N, and each p_i contains S elements.
  • Prompt: The final prompt value summarizes temporal information through the total trend Δ_T, based on consecutive-timestep increments δv_i = v_i+1 − v_i.
  • Problem Definition: Given historical observations X_T = {x_t−T+1:t} ∈ R^T×N and prompts P_S, the task is forecasting future multivariate series over M timesteps.The observation at time t is x_t ∈ R^N, and the forecast is bX_M = {b x_t+1:t+M} ∈ R^M×N.

Methodology

TimeCMA combines dual-modality encoding, cross-modality alignment, and multivariate forecasting to obtain robust time-series representations and predict future values. Its alignment retrieves disentangled embeddings from prompt embeddings, while the forecasting module decodes variable dependencies and projects the result.

  • Framework Overview: TimeCMA comprises dual-modality encoding, cross-modality alignment, and time-series forecasting modules.The framework jointly processes time series and prompts before forecasting.
  • Dual-Modality Encoding: The time-series branch treats each variable’s entire series as a token and processes the resulting embeddings with a Pre-LN Transformer encoder.The inverted embedding first normalizes the input with reversible instance normalization to mitigate distribution shift.
  • Dual-Modality Encoding: The LLM branch uses a frozen GPT-2 to generate prompt embeddings, then refines them with a prompt encoder sharing the time-series encoder’s architecture.The tokenizer converts prompts into token IDs before GPT-2 produces prompt embeddings.
  • Cross-Modality Alignment: Cross-modality alignment uses channel-wise similarity retrieval to obtain disentangled, robust time-series embeddings from entangled, robust prompt embeddings.Linear projections form compact query, value, and key embeddings; similarity-weighted aggregation transfers pretrained LLM knowledge.
  • Time-Series Forecasting: The forecasting module feeds aligned embeddings into a multivariate Transformer decoder and projection function to model variable dependencies and produce forecasts.The model applies normalization, masked self-attention, cross-attention, residual connections, projection, and denormalization.
  • Optimization: The overall loss combines prediction loss Lpre with L2 regularization loss Lreg, weighted by λ.The prediction loss is Mean Squared Error.

Experiments · Datasets. · Baselines and Evaluation.

TimeCMA is evaluated on eight real-world datasets against seven baselines spanning five model categories. Performance is measured using MSE and MAE with a test batch size of 1.

  • Datasets.: Experiments use eight datasets: ETTm1, ETTm2, ETTh1, ETTh2, ECL, FRED-MD, ILI, and Weather.
  • Datasets.: The ETTm1, ETTm2, ETTh1, and ETTh2 datasets are attributed to Zeng et al. (2023).
  • Datasets.: ECL, FRED-MD, ILI, and Weather are sourced from Asuncion and Newman (2007), McCracken and Ng (2016), and Wu et al. (2021).
  • Datasets.: Variables with missing values are removed from FRED-MD, which is simplified as FRED.
  • Baselines and Evaluation.: Seven baselines span prompt-based LLMs, time series-based LLMs, Transformer-based models, linear-based methods, and CNN-based methods.
  • Baselines and Evaluation.: The baselines are Time-LLM, UniTime, OFA, iTransformer, PatchTST, Dlinear, and TimesNet.
  • Baselines and Evaluation.: Evaluation uses mean square error (MSE) and mean absolute error (MAE), with the test batch size set to 1.

Main Results. · Ablation Studies of Model Design.

TimeCMA outperforms all baselines across the reported cases, supporting the use of LLMs and inverted embeddings for multivariate forecasting. Ablations show that cross-modality alignment, the LLM branches, and multivariate temporal decoding are essential design components.

  • Main Results.: TimeCMA outperforms all baselines in all reported cases.Table 1 presents the average performance comparisons.
  • Main Results.: LLM-based models perform better than deep learning and linear models, supporting the use of LLMs for multivariate time series forecasting.
  • Main Results.: Inverted embedding is essential for capturing multivariate dependencies.
  • Main Results.: TimeCMA performs better on datasets with more variables because it introduces inverted embedding and multivariate attention.
  • Ablation Studies of Model Design.: Removing cross-modality alignment has the largest impact among model-design variants.The w/o CMA variant replaces CMA with concatenation, showing that similarity-based retrieval is superior to simple concatenation.
  • Ablation Studies of Model Design.: Removing the LLM is the next most impactful ablation, indicating that the LLM-empowered dual branches improve prediction results.
  • Ablation Studies of Model Design.: Removing the multivariate Transformer decoder shows that decoding long-term temporal dependencies between multiple variables is essential for MTSF.

Ablation Studies of Prompt Design.

The prompt-design ablation shows that prompts ending in numerical values generally improve MSE, with Prompt 5 performing best by abstracting the time-series trend.

  • Prompt Design Ablation: Prompts 3, 4, and 5 generally outperform other designs because their last token is a numerical value.The ablation evaluates five prompts on MSE, with different intended functions for the LLM’s last token.
  • Prompt Design Ablation: Prompt 5 achieves the best performance among numerical last-token prompts by abstracting the time-series trend.The five prompts range from capturing frequency to summarizing trend.
  • Prompt Design Ablation: Prompt 2 emphasizes historical time information.

Model Efficiency Analysis. · Last Token Attention Analysis.

TimeCMA improves efficiency by processing and storing only the last token, reducing training parameters and memory usage. Attention analysis shows that this token focuses most strongly on time-series values, capturing their information effectively.

  • Model Efficiency Analysis.: TimeCMA uses fewer training parameters and less memory than the evaluated alternatives through last-token-only processing and storage.The efficiency comparison uses batch size 8 for fairness; UniTime is excluded from efficiency comparison because it was trained on all datasets.
  • Model Efficiency Analysis.: UniTime has the largest parameter count, whereas Time-LLM has the highest memory usage in the efficiency comparison.The comparison includes TimeCMA, Time-LLM, and OFA under a training batch size of 8.
  • Model Efficiency Analysis.: TimeCMA stores last-token embeddings to accelerate inference and reduce computational costs from long textual prompts.Only the last token is passed to downstream prediction.
  • Last Token Attention Analysis.: The attention analysis segments prompt words and time-series values to measure the final GPT-2 layer’s last-token attention across segments.The analyzed token is the last token < ∆T >.
  • Last Token Attention Analysis.: The last token directs its highest attention toward time-series values, indicating that it captures their value information effectively.This result is shown in the last-token attention visualization in Figure 5.
  • Last Token Attention Analysis.: Five prompts are presented with different purposes for triggering the last token.These prompts are illustrated in Figure 4.

Encoder Attention Analysis.

The encoder attention analysis shows that the Transformer and LLM capture complementary information about multivariable interrelations. Transformer attention is local and variable-specific, whereas LLM textual attention is described as universa.

  • Encoder Attention Analysis.: Transformer and LLM attention capture complementary information about multivariable interrelations, with Transformer attention local and variable-specific and LLM textual attention universa.The analysis visualizes variable attention maps from the time-series and prompt encoders, respectively.

T-SNE Visualization.

T-SNE visualizations show dataset-specific clustering in Transformer time-series embeddings, more complex relations in prompt embeddings, and cohesive cross-modality embeddings. Retrieved embeddings produce well-separated forecasted time-series clusters across datasets, supporting effective forecasting.

  • Transformer time-series embeddings cluster by dataset, indicating that they capture dataset-specific characteristics.
  • Prompt embeddings exhibit more complex inter-relations than time-series embeddings.
  • Cross-modality time-series embeddings are tightly integrated with higher similarity, making retrieved embeddings more cohesive.
  • Forecasted time-series embeddings form well-separated clusters for each dataset, suggesting effective use of retrieved embeddings for accurate forecasts.

Conclusion

TimeCMA is an LLM-empowered multivariate time series forecasting framework that uses cross-modality alignment to aggregate time series and LLM branches, while leveraging the last token embedding to reduce computational costs and accelerate inference.

  • Conclusion: TimeCMA aggregates time series and LLM branches through channel-wise similarity retrieval in a cross-modality alignment module to enhance forecasting.The framework is presented as an LLM-empowered approach for multivariate time series forecasting.
  • Conclusion: Using only the last token embedding reduces computational costs and accelerates inference for the LLM-based forecasting method.The conclusion identifies last-token usage as a promising efficiency mechanism.
Loading 2406.01638v5…