PaperlayerView exact revision on arXiv

Source-linked AI summary

CALF: Aligning LLMs for Time Series Forecasting via Cross-modal Fine-Tuning

Peiyuan Liu, Hang Guo, Tao Dai, Naiqi Li, Jigang Bao, Xudong Ren, Yong Jiang, Shu-Tao Xia

arXiv:2403.07300v3cs.LGcs.CL

TL;DR

LLM-based time-series forecasting is limited by distribution discrepancies between temporal and textual inputs, especially when adapting language models to scarce temporal data. CALF uses cross-modal fine-tuning to align inputs, intermediate features, and outputs, and reports state-of-the-art long- and short-term forecasting with favorable few-shot and zero-shot performance.

  • Problem

    Existing LLM-based forecasting methods neglect distribution discrepancies between textual and temporal input tokens, limiting their effectiveness for time-series forecasting.

  • Method

    CALF uses temporal target and textual source branches with cross-modal matching, feature regularization, and output consistency losses to align modalities.

  • Results

    CALF achieves state-of-the-art performance on long- and short-term forecasting, with favorable few-shot and zero-shot abilities and low computational complexity.

  • Takeaways & Limitations

    Multi-level alignment enables CALF to transfer LLM capabilities to time-series forecasting across standard and limited-data settings.

Abstract

from arXiv · show

Deep learning (e.g., Transformer) has been widely and successfully used in multivariate time series forecasting (MTSF). Unlike existing methods that focus on training models from a single modal of time series input, large language models (LLMs) based MTSF methods with cross-modal text and time series input have recently shown great superiority, especially with limited temporal data. However, current LLM-based MTSF methods usually focus on adapting and fine-tuning LLMs, while neglecting the distribution discrepancy between textual and temporal input tokens, thus leading to sub-optimal performance. To address this issue, we propose a novel Cross-Modal LLM Fine-Tuning (CALF) framework for MTSF by reducing the distribution discrepancy between textual and temporal data, which mainly consists of the temporal target branch with temporal input and the textual source branch with aligned textual input. To reduce the distribution discrepancy, we develop the cross-modal match module to first align cross-modal input distributions. Additionally, to minimize the modality distribution gap in both feature and output spaces, feature regularization loss is developed to align the intermediate features between the two branches for better weight updates, while output consistency loss is introduced to allow the output representations of both branches to correspond effectively. Thanks to the modality alignment, CALF establishes state-of-the-art performance for both long-term and short-term forecasting tasks with low computational complexity, and exhibiting favorable few-shot and zero-shot abilities similar to that in LLMs. Code is available at https://github.com/Hank0626/LLaTA.

Introduction

LLM-based forecasting can help address limited temporal data, but existing methods overlook deeper alignment between textual and temporal representations. CALF addresses this gap through multi-level cross-modal fine-tuning and reports strong forecasting performance.

  • Introduction: Existing LLM-based forecasting methods inadequately align temporal tokens with textual tokens because they primarily address the input side.The paper identifies this distribution discrepancy as a source of sub-optimal performance.
  • Introduction: CALF uses temporal target and textual source branches to enable more comprehensive alignment between time-series information and pretrained LLM information.The temporal branch processes time series, while the textual branch adapts pretrained LLM information from aligned textual tokens.
  • Introduction: Its cross-modal fine-tuning design includes input alignment, feature regularization, and output consistency mechanisms.These components are presented as the framework’s three principal techniques for reducing modality gaps.
  • Introduction: CALF achieves state-of-the-art performance on long- and short-term forecasting across eight real-world datasets, with favorable generalization and low computational complexity.The framework is evaluated across both forecasting horizons rather than a single task setting.

Related Work

Time-series forecasting research has progressed from Transformer-based modeling toward LLM integration and cross-modal transfer. Related methods adapt pretrained language representations, but their alignment strategies remain insufficiently effective.

  • Related Work: Transformer-based models became prominent in forecasting because they model complex dependencies in time-series data.The related-work discussion presents them as leading deep-learning approaches before LLM-based methods.
  • Related Work: LLM-based forecasting adapts pretrained language models to improve context modeling for time-series analysis.Prior work includes unified time-series analysis, soft prompts, and text-prototype reprogramming.
  • Related Work: Existing LLM-based time-series alignment strategies, including soft prompts and text-prototype reprogramming, have not been sufficiently effective.These approaches are described as refinements of the initial LLM-for-time-series paradigm, but their alignment remains limited.
  • Related Work: Cross-modal fine-tuning transfers models from data-rich modalities to data-scarce modalities to address data insufficiency and poor generalization.Previous applications span vision, audio, and biology.

Methodology

CALF uses textual source and temporal target branches with shared pretrained LLM layers, aligning their modalities through input matching, feature regularization, and output consistency. Parameter-efficient fine-tuning trains the temporal branch while inference uses only its output.

  • Two-Branch Architecture: CALF processes aligned text tokens in a textual source branch and projected time-series tokens in a temporal target branch using identical pretrained LLM weights.The two branches produce hidden features and outputs for their respective modalities.
  • Cross-Modal Match Module: The Cross-Modal Match Module aligns temporal tokens with LLM word-embedding distributions using self-attention, PCA-reduced principal embeddings, and cross-attention.PCA reduces the word-embedding dictionary before cross-attention uses principal embeddings as keys and values and temporal tokens as queries.
  • Feature Regularization Loss: Feature regularization aligns intermediate features from corresponding Transformer layers in a shared representation space to guide gradients and weight updates.Trainable projection layers transform textual and temporal features before similarity-based alignment.
  • Output Consistency Loss: Output consistency aligns textual and temporal branch outputs to maintain a coherent semantic representation across modalities.The loss compares the outputs produced by the textual source and temporal target branches.
  • Parameter-Efficient Training: Parameter-efficient fine-tuning applies LoRA and positional-encoding updates to the temporal target branch, while inference uses only its output.This strategy is used to avoid catastrophic forgetting and improve training efficiency.

Experiments

Experiments evaluate CALF on long- and short-term forecasting, few- and zero-shot settings, and computational efficiency across real-world datasets. CALF improves over representative baselines, including PatchTST, TimeLLM, and TimesNet, while maintaining efficient inference.

  • Long-term Forecasting: 7.05%/6.53% lower MSE/MAE than PatchTST and 5.98%/5.34% lower MSE/MAE than TimeLLM demonstrate CALF’s long-term forecasting gains.Results are averaged across prediction lengths, with improvements exceeding 10% against most other baselines.
  • Short-term Forecasting: 1% overall improvement over TimesNet establishes CALF’s superior short-term forecasting performance across the M4 evaluation metrics.The short-term experiments use SMAPE, MSAE, and OWA on horizons from 6 to 48.
  • Few-shot Learning: 8% and 9% average reductions relative to TimeLLM and PatchTST, respectively, show robust performance when only 10% of ETT training data is used.Few-shot experiments are conducted on four ETT datasets.
  • Zero-shot Learning: 4% and 9% improvements over TimeLLM and PatchTST, respectively, demonstrate CALF’s zero-shot transfer across distinct ETT datasets without further training.Zero-shot evaluation trains on one dataset and tests on another.
  • Efficiency Analysis: CALF improves both efficiency and accuracy over other LLM-based methods by using channel sequences as tokens, efficient fine-tuning, and a single time branch during inference.The efficiency comparison covers ETTm1, ECL, Traffic, and Weather with input and prediction lengths both set to 96.

Ablation Study

The ablations examine loss components and PCA dimensionality, showing that modality-alignment losses matter and that performance is relatively robust to the number of principal components.

  • Ablation on Different Loss Functions: The loss ablation evaluates feature regularization, output consistency, and supervised losses on ETTh1 and Weather datasets.The supervised loss directly uses ground-truth data, while the other losses align intermediate features and outputs across modalities.
  • Ablation on the Number of Principal Components: Using d = 500 retains 88% explainable variance while achieving satisfactory performance.Smaller d loses key information, whereas larger d introduces redundancy and learning difficulty.
  • Ablation on the Number of Principal Components: Performance is not highly sensitive to the number of principal components, although very small dimensions degrade results.

Discussion

CALF addresses incomplete alignment in prior LLM-based forecasting methods by aligning temporal inputs with textual representations through cross-attention on word embeddings.

  • Difference from Other Work: Prior methods use input-side cross-attention or contrastive learning, but the paper characterizes these schemes as producing only partial alignment and weak generalization.
  • Interpretability on Implicit Input Alignment: CALF performs cross-attention on word-embedding weights to generate aligned text tokens rather than relying on intuitive natural language.
  • Interpretability on Implicit Input Alignment: Cross-attention maps relate each time-series channel to selected time-related and general words for ETTh1 and ETTh2.Rows represent time-series instances, columns represent selected words, and cells encode channel-word relevance.

Conclusion

CALF bridges temporal and textual distributions for LLM-based time-series forecasting through cross-modal matching and two alignment losses. Experiments report strong forecasting performance, generalization, and low computational complexity.

  • Conclusion: CALF combines a Cross-Modal Match Module, Feature Regularization Loss, and Output Consistency Loss to bridge temporal-textual distribution discrepancies.The framework leverages LLM capabilities for time-series forecasting.
  • Conclusion: Experiments across real-world datasets report a new benchmark in long- and short-term forecasting, with strong generalization and low computational complexity.
Loading 2403.07300v3…