PaperlayerView exact revision on arXiv

Source-linked AI summary

Time-VLM: Exploring Multimodal Vision-Language Models for Augmented Time Series Forecasting

Siru Zhong, Weilin Ruan, Ming Jin, Huan Li, Qingsong Wen, Yuxuan Liang

arXiv:2502.04395v2cs.CVcs.LG

TL;DR

Time series forecasting needs both semantic context and fine-grained temporal information, which existing text- or vision-augmented approaches separately limit. Time-VLM uses retrieval-, vision-, and text-augmented learners with pretrained VLMs to unify these modalities, and experiments report superior performance, especially in few-shot and zero-shot settings. Its limitations include constrained time-series semantics in current VLMs, some full-shot gaps versus specialized unimodal models, and deployment challenges on resource-constrained devices.

  • Problem

    Text-augmented forecasting can lose information across the modality gap and miss fine-grained temporal patterns, while vision-augmented methods lack semantic interpretability; integrating both remains underexplored.

  • Method

    Time-VLM combines a retrieval-augmented learner, a vision-augmented learner, and a text-augmented learner with pretrained VLMs to fuse temporal, visual, and textual embeddings for forecasting.

  • Results

    Time-VLM achieves superior performance across diverse datasets, especially in few-shot and zero-shot settings.

  • Takeaways & Limitations

    Time-VLM establishes a multimodal direction for time series forecasting and is particularly robust where auxiliary data is scarce.

  • Takeaways & Limitations

    Current VLMs constrain TAL’s semantic impact, full-shot performance can lag specialized unimodal models on some tasks, and resource-constrained deployment remains challenging.

Abstract

from arXiv · show

Recent advancements in time series forecasting have explored augmenting models with text or vision modalities to improve accuracy. While text provides contextual understanding, it often lacks fine-grained temporal details. Conversely, vision captures intricate temporal patterns but lacks semantic context, limiting the complementary potential of these modalities. To address this, we propose \method, a novel multimodal framework that leverages pre-trained Vision-Language Models (VLMs) to bridge temporal, visual, and textual modalities for enhanced forecasting. Our framework comprises three key components: (1) a Retrieval-Augmented Learner, which extracts enriched temporal features through memory bank interactions; (2) a Vision-Augmented Learner, which encodes time series as informative images; and (3) a Text-Augmented Learner, which generates contextual textual descriptions. These components collaborate with frozen pre-trained VLMs to produce multimodal embeddings, which are then fused with temporal features for final prediction. Extensive experiments demonstrate that Time-VLM achieves superior performance, particularly in few-shot and zero-shot scenarios, thereby establishing a new direction for multimodal time series forecasting. Code is available at https://github.com/CityMind-Lab/ICML25-TimeVLM.

1. Introduction

Time-VLM addresses the complementary limitations of text- and vision-augmented forecasting by unifying temporal, visual, and textual modalities with pretrained VLMs. Its retrieval-, vision-, and text-augmented learners support forecasting, with evaluations reporting especially strong performance in data-scarce settings.

  • Text augmentation supplies semantic context but can lose information across the continuous-time-series/discrete-text modality gap and miss fine-grained temporal dynamics.
  • Vision augmentation captures spatial and hierarchical temporal patterns but lacks semantic interpretability and domain-specific knowledge.
  • Time-VLM unifies temporal, visual, and textual modalities through pretrained VLMs rather than relying on single-modality projections.
  • Its Retrieval-Augmented Learner enhances temporal features through patch extraction and memory-bank interactions, while the Vision- and Text-Augmented Learners generate image and textual representations.
  • Extensive evaluations report strong Time-VLM performance, especially under data-scarce conditions.

2. Related Work

Related work has separately applied language and vision models to time series forecasting, while VLMs align image and text representations. Time-VLM extends this direction by integrating temporal, visual, and textual modalities for forecasting.

  • Text-Augmented Models for Time Series Forecasting: Text-augmented methods map time series into textual representations or tokenize them for autoregressive prediction, but face modality-gap, arithmetic, and recursive-reasoning limitations.
  • Vision-Augmented Models for Time Series Forecasting: Vision-augmented methods transform time series into matrices or images and use CNNs or pretrained visual encoders to preserve temporal patterns.
  • VLMs align image and text representations for multimodal understanding, but remain underexplored for time series analysis.
  • Time-VLM integrates temporal, visual, and textual modalities to address the limitations of unimodal approaches.

3. Methodology

Time-VLM integrates retrieval-based temporal features with vision- and text-derived context through frozen VLMs for time series forecasting. Its learners encode historical patterns, informative images, and textual descriptions, then fuse these modalities for prediction with lightweight optimization.

  • Framework overview: Time-VLM combines Retrieval-, Vision-, and Text-Augmented Learners to integrate temporal, visual, and textual information for forecasting.The multimodal fusion pipeline uses VLM-derived embeddings alongside temporal memory features.
  • Multimodal fusion and optimization: Generated images and text are encoded by frozen VLMs, fused with temporal memory features through a gated mechanism, and optimized with lightweight components.The frozen VLM is retained while the prediction head uses gating and linear projection to generate forecasts.
  • Retrieval-Augmented Learner: The Retrieval-Augmented Learner divides sequences into overlapping patches and enriches their embeddings through hierarchical memory retrieval and self-attention.Historical patch representations are stored in a circular-buffer memory bank, while local and global memories capture complementary dependencies.
  • Vision-Augmented Learner: The Vision-Augmented Learner converts time series into images using frequency and periodicity encodings, multi-scale convolutions, interpolation, and normalization.These operations preserve fine-grained details and high-level temporal structures before visual encoding.
  • Vision-Augmented Learner: Multi-scale convolutions extract hierarchical temporal patterns before resizing the resulting tensor into normalized image representations for a VLM vision encoder.The convolutional stages capture local dependencies and produce output channels for the final image representation.
  • Text-Augmented Learner: The Text-Augmented Learner builds prompts from statistics, trends, periodicity, forecasting parameters, domain context, and image descriptions.The resulting textual embeddings complement visual and temporal features and support both static and dynamically generated context.

4. Experiments

Experiments evaluate Time-VLM across few-shot, zero-shot, short-term, and long-term forecasting, together with ablations, efficiency, hyperparameter, and VLM-backbone analyses. Results show strong accuracy, cross-domain transfer, and parameter efficiency, while revealing that retrieval and vision components contribute more than textual augmentation.

  • Experimental setup: Evaluations cover seven time-series datasets, the M4 short-term benchmark, and comparisons against text-, vision-, traditional-, and deep-learning baselines.The unified evaluation pipeline uses the same configurations as prior work for fair comparison.
  • Few-shot forecasting: 29.5% lower MSE and 16.6% lower MAE than TimeLLM on ETTh1 with 5% training data demonstrate strong few-shot performance.On ETTm1 with 10% data, improvements are 11.1% in MSE and 10.5% in MAE; on Weather with 5% data, they are 7.7% and 9.4%.
  • Zero-shot forecasting: 4.2% lower MSE and 0.5% lower MAE than TimeLLM in ETTh1->ETTh2 show effective zero-shot cross-domain transfer without fine-tuning.The model also exceeds TimeLLM by 7.1% in MSE and 3.6% in MAE for ETTm1->ETTh2, while closely matching it for ETTm2->ETTh2.
  • Short-term forecasting: 0.7% improvements in SMAPE, 0.2% in MASE, and 0.5% in OWA over Time-LLM establish strong short-term M4 performance with fewer resources.Time-VLM consistently outperforms state-of-the-art baselines across all reported metrics.
  • Long-term forecasting: Time-VLM improves over TimeLLM on ETTh1 and ETTm2 but underperforms it on Weather, indicating competitive yet dataset-dependent long-term forecasting.The Weather differences are 0.4% higher MSE and 2.3% higher MAE.
  • Model analysis: Removing RAL increases MSE by 35.6% and removing VAL by 9.0%, whereas removing TAL causes only 2.1% degradation, highlighting unequal component contributions.Time-VLM uses 143M parameters versus TimeLLM’s 3405M, and the VLM analysis finds that larger backbones do not necessarily improve accuracy.

5. Conclusion

Time-VLM unifies temporal, visual, and textual modalities through RAL, VAL, and TAL, using pretrained VLMs for time series forecasting. It performs strongly across datasets, especially in few-shot and zero-shot settings, but has limitations in semantic understanding, full-shot performance, and deployment on resource-constrained devices.

  • Framework and performance: Time-VLM integrates retrieval-, vision-, and text-augmented learners with pretrained VLMs to unify temporal, visual, and textual modalities.The framework operates solely on raw time series data and generates textual and visual representations internally.
  • Framework and performance: Time-VLM achieves superior performance across diverse datasets, particularly in few-shot and zero-shot settings, while maintaining computational efficiency.The conclusion reports that it outperforms existing methods in these settings.
  • Limitations: The TAL module has limited impact because current VLMs have constrained understanding of time series semantics.The authors identify temporally aware VLMs and stronger temporal reasoning as directions for improvement.
  • Limitations: Full-shot performance slightly lags behind specialized unimodal models on certain tasks, including ECL and Traffic.This suggests room for domain-specific adaptation.
  • Limitations: Deployment on resource-constrained devices remains challenging despite being computationally efficient compared with LLM-based methods.The authors propose visual distillation and related efficiency improvements as future directions.

Impact Statement

The paper aims to advance machine learning by integrating temporal, visual, and textual modalities for time series forecasting. It acknowledges privacy, bias, and computational-cost risks relevant to responsible deployment in high-stakes applications.

  • Goal: The work integrates temporal, visual, and textual modalities to advance time series forecasting.The impact statement frames this integration as the paper’s machine-learning goal.
  • Risks: Potential risks include data privacy concerns, algorithmic bias, and increased computational costs.The authors encourage further research to mitigate these risks before responsible deployment in high-stakes applications.

A.1. Dataset Details

The experiments use benchmark datasets spanning temperature, electricity, transportation, weather, and forecasting-competition domains. These datasets vary in sampling frequency, sequence length, periodicity, and train-validation-test structure.

  • Dataset coverage: The benchmark suite covers temperature monitoring, electricity consumption, transportation, weather forecasting, and M4 competition data.Datasets include ETT, ECL, Traffic, Weather, and M4.
  • Dataset structure: Datasets contain multiple time series with varying sequence lengths and are divided into training, validation, and testing sets.Collection frequencies range from 15 minutes to yearly intervals.
  • Dataset details: ECL records one-minute household electricity consumption over four years, totaling 2,075,259 measurements.The data come from a house in Sceaux, France, and include electrical quantities and sub-metering values.
  • Dataset details: ETTh1 and ETTh2 provide hourly ETT data, while ETTm1 provides 15-minute data with oil temperature and six power-load features.The ETT subsets use a 12/4/4 month train-validation-test split.
  • Periodicity encoding: Periodicity P encodes dominant temporal cycles, such as daily patterns, and the resulting encodings are concatenated with the input time series.Examples include P = 96 for 15-minute ETT data, P = 24 for hourly ETT data, and P = 144 for 10-minute Weather data.

A.2.1. MODEL ARCHITECTURE PARAMETERS

Time-VLM uses compact image representations, a 128-dimensional model backbone, a shallow encoder-decoder, dropout, and a gated multimodal fusion module. Its VLM produces 156-token embeddings with hidden dimension 768.

  • Core architecture: Time-VLM represents images at 64 × 64 and uses a backbone hidden dimension of d_model = 128.The image size balances computational efficiency and temporal information preservation.
  • Core architecture: The encoder-decoder contains e_layers = 2 encoder layers and d_layers = 1 decoder layer, with dropout = 0.1.The dropout rate is applied to mitigate overfitting.
  • Multimodal fusion: The gated fusion module uses d_fusion = 256, while the VLM produces vlm_fused_len = 156 tokens with vlm_hidden_dim = 768.These dimensions support integration with the pretrained VLM architecture.

A.2.2. TRAINING PARAMETERS

The framework uses a comprehensive training strategy with specified optimization settings, early stopping, forecasting horizons, and evaluation metrics for long- and short-term tasks.

  • Training uses batch size 32, initial learning rate 0.001, AdamW, early stopping patience 3, MSE loss, and a maximum of 10 epochs.
  • The input sequence length is 512, with prediction lengths beginning at 96.
  • Long-term forecasting is evaluated with MSE and MAE, while M4 short-term forecasting uses SMAPE, MASE, and OWA.
  • The metric definitions use H for prediction horizon, s for periodicity, and Yh and Ŷh for ground truth and prediction at time step h.

B. Complete results

The complete-results materials cover few-shot, zero-shot, short-term, and long-term forecasting, alongside image representations designed to preserve temporal characteristics.

  • Complete results: Few-shot results cover 5% and 10% training data, with horizons H ∈{96, 192, 336, 720} for the 5% setting and lower values indicating better performance.
  • Complete results: Zero-shot results are reported on ETT datasets, with lower values indicating better performance.
  • Complete results: Short-term results use horizons in [6, 48], including weighted averages across datasets and sampling intervals.
  • Complete results: Long-term forecasting results follow the protocol used in Table 1.
  • Image representations: The image-generation module combines frequency and periodicity encoding, multi-scale convolution, interpolation, and normalization.
  • Image representations: Generated images represent frequency characteristics through textures and encode multi-scale temporal dependencies through regular visual patterns.
  • Image representations: Color intensity represents values, while abrupt color transitions highlight sudden changes or anomalies.

C.2. Visualization of prediction results

Visualization results compare true and predicted values across ETTh1, ETTm1, ECL, and Traffic at multiple horizons, showing strong alignment on periodic datasets and deviations on irregular Traffic patterns.

  • Prediction visualizations: Figures 7, 8, 9, and 10 visualize true values as solid lines and predictions as dashed lines across four datasets and multiple horizons.
  • Periodic datasets: Time-VLM captures global trends and fine-grained temporal patterns effectively on datasets with clear periodic structures, including ETTh1 and ETTm1.
  • Irregular datasets: Traffic shows slight deviations in sudden fluctuations, particularly at longer horizons of 336 and 720.
  • Irregular datasets: The Traffic deviations suggest opportunities to refine the time series-to-image transformation for highly irregular data.

D.1. Limitations

The framework has limitations on irregular data and in its use of general-purpose pretrained VLMs, motivating improved visual transformations, larger models, interpretability, and domain-specific pretraining.

  • Limitations: Time-VLM performs less robustly on highly volatile or irregular patterns than on datasets with periodic structures.
  • Limitations: Current visual transformations may not adequately capture abrupt temporal dynamics or sudden shifts.
  • Limitations: The framework relies on ViLT and CLIP, which are optimized for natural vision-language tasks rather than time series forecasting.
  • Limitations: These VLMs have limited textual capabilities, shorter text-input support, and limited domain-specific time-series knowledge.
  • Future work: Future work includes adaptive visual transformations, larger multimodal VLMs, and analysis of trade-offs among model size, efficiency, and performance.
  • Future work: Future research could study multimodal feature contributions and develop time-series-specific foundation models using paired temporal, textual, and visual data.
Loading 2502.04395v2…