Source-linked AI summary
SAGE: Variate-Wise Semantic Augmentation for Vision-Language Time Series Forecasting
Haizhao Fan, Xinyi Le
TL;DR
Time-series forecasting lacks variable-level semantic knowledge, while existing multimodal approaches may depend on costly LLM inference or use uniform textual representations. SAGE integrates CLIP-based temporal, textual, visual, and cross-variable modeling with gated variable-specific knowledge injection, achieving strong results across long-term benchmarks and M4 without an LLM in the forecasting loop.
Problem
Existing multimodal forecasting methods do not fully exploit CLIP’s jointly aligned text and vision encoders and may require expensive LLM inference or limited textual representations.
Method
SAGE adapts CLIP’s text encoder for frequency-enhanced temporal modeling, uses its frozen vision encoder for training-only contrastive supervision, and injects variable-specific descriptions and statistics through gated paths.
Results
SAGE achieves the best average MSE on 7 of 8 long-term datasets, the best average MAE on 6 of 8, and the best M4 OWA of 0.834.
Takeaways & Limitations
Aligned temporal, relational, textual, and visual evidence can improve a compact forecasting backbone, while text enhancement improves 31 of 32 dataset-horizon settings.
Takeaways & Limitations
The method relies on hand-crafted templates, text benefits diminish on high-dimensional datasets, and descriptions omit external event or ontology information.
Abstract
from arXiv · showhide
Time series forecasting models operate on raw numerical sequences, lacking the semantic knowledge that domain experts implicitly leverage, such as the physical meaning of each variable, its statistical behavior, and its temporal dynamics. Recent efforts to bridge this gap fall into two camps. Some rely on large language models at inference time, which is computationally expensive. Others apply uniform textual prompts at the dataset level, ignoring the heterogeneous semantics across individual variates. We propose SAGE (Seeing and Augmenting with Grounded Encoding), an end-to-end CLIP-based framework that jointly models temporal, cross-variable, textual, and visual information. The CLIP text encoder processes frequency-enhanced patches and variable tokens, while gated residual paths inject variable-specific descriptions and statistical descriptors. In parallel, the frozen CLIP vision encoder aligns rendered series with temporal representations through a training-only contrastive objective. This dual use of CLIP adds complementary semantic and visual supervision without placing an LLM in the forecasting loop. Across eight long-term benchmarks and M4, SAGE achieves state-of-the-art accuracy. Ablations confirm complementary gains from multimodal alignment and variable-level knowledge.
1 Introduction
SAGE addresses limitations in multimodal time-series forecasting by using CLIP’s text and vision encoders together with variable-specific knowledge injection. Its design combines temporal, cross-variable, textual, and visual signals without requiring an LLM during forecasting.
- Research gap: Existing multimodal methods often use language models as frozen feature extractors or auxiliary prompt generators, leaving CLIP’s dual vision-language structure underused.They may also require expensive LLM inference or reduce numerical sequences to limited textual representations.
- SAGE framework: SAGE makes CLIP part of the forecasting architecture by adapting its text encoder as the temporal backbone and using its frozen vision encoder for training-time contrastive supervision.This provides a compact trainable backbone instead of relying on a billion-parameter model in the forecasting loop.
- Empirical preview: Figure 1 presents representative 96-step forecasts on Weather, Electricity, and ETTm1, with inset window-level MSE comparisons against the strongest displayed baseline.The figure highlights SAGE’s ability to capture regime changes using numerical history and multimodal knowledge.
- SAGE framework: Variable-wise gated cross-attention injects offline semantic, behavioral, and relational descriptions, while a separately gated statistical bypass preserves numerical cues.An LLM may assist description preparation, but is not called during training or forecasting.
- SAGE framework: Frequency-enhanced patch embeddings combine time-domain and spectral evidence, while cross-variate attention models dependencies among variables.These temporal and relational representations are further regularized through vision-language contrastive alignment.
2 Related Work
Related work spans language-model backbones, cross-modal alignment, multimodal fusion, supervised temporal architectures, and contrastive representation learning. These approaches motivate SAGE’s use of transferable vision-language representations and multimodal temporal modeling.
- Language and cross-modal methods: Early methods use pretrained language models as time-series backbones, encode numerical series as text, or align time-series embeddings with text prototypes.Examples include GPT4TS, LLMTime, Time-LLM, and TEST.
- Multimodal fusion: Recent methods move toward multimodal fusion by combining retrieval, vision, and text, supporting zero-shot forecasting or anomaly detection with frozen vision-language models.Examples include Time-VLM, Aurora, and VLM4TS.
- Supervised forecasting: Supervised forecasting architectures model decomposition, channel-independent patches, variate tokens, or two-dimensional temporal structure to capture temporal and cross-variable patterns.Representative methods include Autoformer, PatchTST, iTransformer, and TimesNet.
- Pretraining trade-offs: Large-scale pretraining can yield transferable temporal knowledge, but it requires curating massive time-series corpora.OccamVTS instead suggests leveraging pretrained vision-language representations without billion-scale pretraining.
- Contrastive learning: Contrastive learning methods improve transferable time-series representations through temporal-context augmentation, hierarchical objectives, time-frequency losses, and consistency constraints.These methods establish contrastive learning as a relevant foundation for SAGE’s alignment objective.
3 Methodology
SAGE uses a shared CLIP text encoder to model temporal and cross-variable structure, then injects variable-specific semantic and statistical knowledge through gated fusion. A frozen CLIP vision encoder adds training-only contrastive supervision before forecast decoding and inverse normalization.
- Frequency-Enhanced Language Module: SAGE normalizes each variable, divides its sequence into overlapping patches, and maps patches into CLIP’s shared embedding width.A learnable tokenizer produces patch tokens, while RevIN stores per-variable statistics for later de-normalization.
- Frequency-Enhanced Language Module: Frequency-enhanced temporal tokens combine time-domain patches with Hann-windowed FFT features through cross-attention controlled by a learnable scalar.The time token supplies the query, and the frequency token supplies keys and values.
- Cross-Variable Modeling: The same CLIP text encoder processes variable tokens to capture cross-variable structure alongside independently encoded temporal representations.Temporal summaries query contextual variable representations through cross-attention and residual updates.
- Multi-View Textual Semantic Fusion Module: Variable-specific descriptions are built offline from metadata and training statistics, embedded by CLIP, and fused through learnable per-variable gates.Templates may include units, domain roles, distributional statistics, trend slope, dominant FFT period, and correlations with other variables.
- Multi-View Textual Semantic Fusion Module: Statistical descriptors bypass attention through a separate residual path, complementing soft textual semantics with hard numerical evidence.A two-layer network maps each 12-dimensional statistical vector into the shared embedding width.
- Vision-Language Contrastive Alignment Module: For datasets with at most 50 variables, rendered variable sequences receive frozen-vision CLIP contrastive supervision during training; larger datasets disable this loss for practicality.The fused representations are decoded into forecast steps and inverse-normalized, while textual fusion remains active for N > 50.
4 Experiments
SAGE is evaluated across diverse long- and short-term forecasting benchmarks using fixed lookback settings, multiple metrics, and systematic text and vision ablations. It achieves strong average accuracy, while the ablations show complementary benefits from textual enhancement and vision–temporal alignment, with dataset-dependent limitations.
- 4.1 Setup: SAGE is evaluated on eight long-term benchmarks and the M4 short-term competition using MSE, MAE, SMAPE, MASE, and OWA.The long-term experiments use T=96 with prediction horizons of 96, 192, 336, and 720.
- 4.2 Main Results: SAGE obtains the best average MSE on 7 of 8 long-term datasets and the best average MAE on 6 of 8, with overall averages of 0.331 MSE and 0.348 MAE.The gains span datasets ranging from 8 to 862 variables, and offline text preparation avoids LLM calls during training or forecasting.
- 4.2 Main Results: SAGE leads all three M4 metrics with an OWA of 0.834, ahead of TimesNet at 0.851 and N-HiTS at 0.861.This extends the reported performance beyond long-horizon forecasting to the short-term M4 benchmark.
- 4.3 Text Enhancement Analysis: Text enhancement improves forecasting in 31 of 32 dataset-horizon combinations, producing an average MSE reduction of 2.8%.+Stat is the most reliable mode, while +Stat+Dyn reaches the largest reported peak improvement of 6.3% on Exchange.
- 4.3 Text Enhancement Analysis: Text gains tend to be larger on lower-dimensional datasets, including 6.3% on Exchange (N=8), 3.9% on Weather (N=21), 1.2% on ECL (N=321), and 0.9% on Traffic (N=862).The relationship is not strictly monotonic, and per-variable gates can suppress unhelpful textual information on high-dimensional data.
- 4.4 Vision Enhancement Analysis: Vision–temporal alignment improves forecasting on all six eligible datasets, reducing average MSE from 0.348 to 0.343, or 1.6% relatively.The visual branch is active only during training, so the forecast head receives no image features and incurs no visual computation at forecasting time.
5 Conclusion
SAGE combines a dual-use CLIP backbone with variable-specific textual knowledge and frequency-enhanced temporal encoding for multivariate forecasting. It achieves strong benchmark performance, while several limitations remain around text construction, high-dimensional data, external context, and forecasting scope.
- SAGE uses a trainable CLIP text encoder, a frozen vision encoder for training-only contrastive supervision, variable-specific knowledge injection, and frequency-enhanced temporal encoding.
- SAGE achieves the best average MSE on 7 of 8 long-term datasets, the best average MAE on 6 of 8, and the best M4 OWA of 0.834.
- Text enhancement improves 31 of 32 dataset-horizon settings, supporting gains from aligned temporal, relational, textual, and visual evidence in a compact forecasting backbone.
- Current limitations include hand-crafted text templates, diminishing text benefits on high-dimensional datasets, no external event or ontology information, and limited extensions to multivariate-to-multivariate or probabilistic forecasting.
Appendix A: Full Per-Horizon Forecasting Results
Table 7 reports full per-horizon long-term forecasting results using MSE and MAE, comparing representative methods under lookback T=96.
- Table 7 compares representative methods using per-horizon MSE and MAE results under lookback T=96.