PaperlayerView exact revision on arXiv

Source-linked AI summary

From Images to Signals: Are Large Vision Models Useful for Time Series Analysis?

Ziming Zhao, ChengAo Shen, Hanghang Tong, Dongjin Song, Zhigang Deng, Qingsong Wen, Jingchao Ni

arXiv:2505.24030v2cs.LGcs.AIcs.CV

TL;DR

The paper asks whether large vision models are useful for time-series analysis, a question left open despite proposed links between image representations and classification and forecasting. It studies four LVMs, eight imaging methods, and benchmarks across both tasks; it finds LVMs useful for classification but challenged in forecasting.

  • Problem

    Whether LVMs are useful for time-series analysis remains underexplored, despite proposed connections between image representations and both semantic classification and numerical forecasting.

    Source: §1 · p. 1

  • Method

    The study compares four pre-trained LVMs across eight time-series imaging methods and classification and forecasting benchmarks, with ablations of model components and design choices.

    Source: §1 · p. 2

  • Results

    LVMs are useful for time-series classification but face forecasting challenges, and current forecasters are constrained by model and imaging choices, period bias, and limited use of long look-back windows.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    The forecasting analyses highlight a need to improve encoder utilization, address inductive bias, handle longer look-back windows, and diversify benchmarks.

    Source: §5 · p. 9

    Current best LVM forecasters are limited to specific LVM types and imaging methods, exhibit bias toward forecasting periods, and have limited ability to use long look-back windows.

    Source: §1 · p. 2

Abstract

from arXiv · show

Transformer-based models have gained increasing attention in time series research, driving interest in Large Language Models (LLMs) and foundation models for time series analysis. As the field moves toward multi-modality, Large Vision Models (LVMs) are emerging as a promising direction. In the past, the effectiveness of Transformer and LLMs in time series has been debated. When it comes to LVMs, a similar question arises: are LVMs truely useful for time series analysis? To address it, we design and conduct the first principled study involving 4 LVMs, 8 imaging methods, 18 datasets and 26 baselines across both high-level (classification) and low-level (forecasting) tasks, with extensive ablation analysis. Our findings indicate LVMs are indeed useful for time series classification but face challenges in forecasting. Although effective, the contemporary best LVM forecasters are limited to specific types of LVMs and imaging methods, exhibit a bias toward forecasting periods, and have limited ability to utilize long look-back windows. We hope our findings could serve as a cornerstone for future research on LVM- and multimodal-based solutions to different time series tasks.

1 Introduction

The paper asks whether pre-trained Large Vision Models (LVMs) help with time-series analysis, a question that remains underexplored despite proposed links between image representations and both classification and forecasting. It studies LVMs across these tasks and finds them useful for classification but challenged by forecasting.

1 Introduction

  • The paper frames LVMs as candidates for both semantic classification and numerical forecasting because images can encode temporal patterns and sequences of continuous pixel values.Despite these proposed connections, in-depth links between LVMs and time-series analysis remain underexplored.Source: §1 · p. 1
  • Pre-trained LVMs are useful for time-series classification but face forecasting challenges, with current forecasters limited by model and imaging choices, period bias, and long look-back use.The best forecasters also rely on self-supervised LVMs and a specific imaging method, UVH; their decoder components are more critical than their encoders for forecasting.Source: §1 · p. 2
  • The study evaluates four LVMs, eight imaging methods, 10 classification datasets, eight forecasting datasets, and 26 baselines across the two tasks.It compares LVMs with 18 classification and eight forecasting baselines and analyzes pre-training, imaging, decoding, fine-tuning, architecture, temporal order, and computational costs.Source: §1 · p. 2
  • The study compares zero-shot and fine-tuned LVMs with same-architecture models trained from scratch, finding that pre-trained Transformer components transfer useful knowledge.It also tests shuffled time steps and conducts forecasting-specific analyses of model components, inductive bias, and look-back windows.Source: §1 · p. 2

2 Related Work

Prior work applies vision models to several time-series tasks, while this paper focuses on pre-trained large vision models and covers both classification and forecasting. Figure 1 previews the imaging methods and LVM task frameworks considered.

2 Related Work

  • Earlier vision-model applications span time-series classification, forecasting, anomaly detection, and generation, while this work focuses on pre-trained LVMs, particularly Transformer-based models.Earlier CNN applications are outside its scope because of their relatively smaller sizes.Source: §2 · p. 2
  • Figure 1 previews eight imaging methods and three LVM task frameworks, with the encoder-decoder forecasting framework limited to MAE and SimMIM using UVH or MVH images.The figure also distinguishes MVH, which encodes multivariate series, from the other imaging methods, which encode univariate series.Source: Figure 1 · p. 3
  • Prior LVM-based work commonly images time series before feeding them to a model; forecasting has fewer such efforts, with VisionTS adapting MAE for zero-shot and few-shot forecasting.This study adds SimMIM alongside MAE for forecasting.Source: §2 · pp. 2–3
  • Large vision-language models have been explored for classification, anomaly detection, and forecasting, but this paper focuses on sole LVMs and leaves VLMs for future work.The authors state that the effectiveness of sole LVMs in time-series analysis is not yet well understood.Source: §2 · p. 3

3 Methods for Using LVMs in Time Series Analysis

The methods adapt pre-trained LVMs to time series by imaging and aligning inputs, then use task-specific classification or forecasting designs. The paper also defines simpler ablation models to assess the role of LVM architecture.

3 Methods for Using LVMs in Time Series Analysis

  • Time series are imaged, resized to each LVM’s required size and channel count, and normalized before being divided into model-specific patches.The adopted LVMs expect normalized three-channel images of a predefined size.Source: §3 · pp. 3–4
  • Eight imaging methods cover line plots, MVH, UVH, three spectrogram methods, GAF, and RP; only MVH encodes multivariate series directly.UVH stacks period-length segments into an image, while STFT, Wavelet, and Filterbank encode time and frequency as spectrograms.Source: §3 · p. 3
  • For classification, each LVM encoder is linearly probed, with per-variate embeddings concatenated for imaging methods that encode univariate series.MVH supplies a single image of all variates and therefore needs no variate concatenation.Source: §3 · p. 4
  • Forecasting uses either a linear forecaster or an LVM reconstruction decoder, with the decoder framework available only for MAE and SimMIM using UVH or MVH.The decoder framework masks the forecast region and reconstructs masked patches; the linear framework can use all imaging types.Source: §3 · p. 4
  • Both forecasting frameworks assume variate independence except with MVH, where all variates share one image and are forecast together.The decoder framework requires pixel values to preserve raw time-series values, restricting its imaging choices to MVH and UVH.Source: §3 · p. 4
  • The W/O-LVM and LVM2ATTN ablations retain the encoder projection layer but replace its Transformer with a linear layer or one randomly initialized attention layer.Both use a linear head and apply across all imaging types and both tasks.Source: §3 · p. 4

4 Experiments

The experiments compare four LVMs and multiple imaging methods across classification and forecasting benchmarks, then probe transfer, architecture, temporal order, computational cost, and forecasting-specific constraints. LVMs outperform baselines in classification, while forecasting performance depends on model and imaging choices and is constrained by decoder use, periodicity, and look-back length.

4.2 Results of Comparing LVMs with Non-LVM Methods

  • Both ViT and MAE outperform classification baselines; in forecasting, MAE beats non-LVM baselines in 9/16 dataset-metric cases, while ViT shows no evident overall superiority.ViT remains comparable to DLinear in many forecasting cases.Source: Figure 2 · p. 5, Table 2 · p. 5, §4.2 · p. 5

4.3 In-Depth Analysis of LVMs’ Suitability in Time Series Tasks

  • Self-supervised and supervised LVMs have comparable classification accuracy, but self-supervised models forecast better; window-based attention can miss time-sensitive patterns.The paper attributes this weakness to local attention's translation-invariance assumption, which does not hold when image locations represent ordered time steps or frequencies.Source: §4.3 · pp. 5–6, Figure 3 · p. 6

[RQ2] Which imaging method best fits TSC (TSF) task?

  • GAF ranks best for classification, close to MVH and RP, while UVH and MVH rank best for forecasting with reconstruction.UVH and MVH preserve raw values in pixels; other imaging methods use the linear-forecaster framework instead.Source: [RQ2] Which imaging method best fits… · p. 6, Figure 4 · p. 6
  • For classification, pre-training helps and limited fine-tuning can match full fine-tuning; for forecasting, norm-only fine-tuning boosts performance, and LVMs outperform simpler alternatives in key transfer settings.The authors suggest that forecasting's numerical outputs are normalization-sensitive, whereas classification benefits from learning across more layers.Source: Table 3 · p. 6, [RQ2] Which imaging method best fits… · p. 7
  • Temporal perturbations reduce LVM performance more than that of the compared ablations, suggesting LVMs use temporal patterns during inference.The four perturbations include shuffling all or half the time points, swapping halves, and masking half the points.Source: [RQ2] Which imaging method best fits… · p. 7, Table 4 · p. 7
  • LVMs take 3x and 16x the training time of the best classification and forecasting baselines, respectively; they are 4x faster for classification inference but 20x slower for forecasting inference.For forecasting, the authors attribute the overhead to model size and the extra cost of imaging time series.Source: [RQ2] Which imaging method best fits… · p. 8, Table 5 · p. 8, Figure 6 · p. 8

4.4 More Analysis of LVMs’ Suitability in Time Series Forecasting

  • In forecasting, pre-trained decoders contribute more than encoders, and UVH makes MAE perform best at segment lengths L and 2L, indicating a bias toward forecasting periods.Even SimMIM's linear decoder, with only 3.8% of its parameters, can outweigh its much larger encoder in this task.Source: §4.4 · pp. 8–9
  • MAE improves with longer look-back windows up to length 1000, then plateaus or declines; recurrence difficulty aligns with errors on ETTh1 and ETTm1 but less on Weather and Illness.The authors suggest fixed-size image inputs may compress long series and lose temporal information; weaker periodic patterns may explain the weaker alignment on Weather and Illness.Source: §4.4 · p. 9

5 Conclusion

The paper examines image-pretrained LVMs for classification and forecasting, using experiments and ablations to identify where they help and where forecasting remains constrained.

5 Conclusion

  • The study evaluates LVMs on both classification and forecasting tasks and uses ablations to assess how image pre-training benefits them.It aims to inform adoption across research and practical applications.Source: §5 · p. 9
  • Current forecasting analysis highlights needs to improve encoder utilization, address inductive bias, handle longer look-back windows, and diversify benchmarks.Source: §5 · p. 9
  • The authors present the study as groundwork for multimodal, agentic time-series analysis.Source: §5 · p. 9

A.1 Benchmarks

The appendix specifies the classification and forecasting benchmarks, baselines, evaluation and implementation protocols, and imaging procedures. It also identifies where full benchmark results are reported, without stating their comparative values.

A.1 Benchmarks

  • The TSC benchmarks comprise 10 multivariate UEA datasets, while TSF uses 8 widely used forecasting datasets.The TSC datasets span several domains; TSF includes ETT, Weather, Illness, Traffic, and Electricity data.Source: §A.1 · p. 14, Table 6 · p. 14, Table 7 · p. 14

A.2 Baselines

  • Following [65], comparison includes 18 conventional and SOTA baselines for TSC; following [41] [5], it includes 8 representative LLM-, Transformer-, and non-Transformer-based TSF baselines, totaling 21 distinct baselines across tasks.Source: §A.2 · p. 14

A.3 Evaluation Metrics

  • Classification is evaluated by accuracy, and forecasting by MSE and MAE over predictions across variates and future time steps.The TSF protocol generally uses look-back length H = 336 and prediction lengths T ∈ {96, 192, 336, 720}; Illness uses H = 104 and T ∈ {24, 36, 48, 60}.Source: §A.3 · p. 15

A.4 Implementation Details

  • The implementation uses four pretrained LVM checkpoints, Adam optimization, task-specific batch sizes and epoch limits, and early stopping.Experiments are repeated three times and averaged, using NVIDIA RTX 6000 Ada Generation GPUs.Source: §A.4 · p. 16

A.5 Imaging Methods

  • GAF maps normalized time-series values to polar coordinates and encodes pairwise temporal correlations, while UVH reshapes a padded series using a dominant-frequency-derived period.UVH's segment lengths are selected using FFT on a 1152-step look-back window, except for Illness, which uses 104 steps.Source: §A.5 · p. 16

B.1 Full Results of Time Series Classification

  • Full TSC accuracy results across 10 benchmark datasets are reported, with LVM standard deviations provided separately.The supplied passages identify the results table but do not provide its cell values.Source: §B.1 · p. 17, Table 9 · p. 17, Table 10 · p. 17

B.2 Full Results of Time Series Forecasting

  • Full TSF MSE and MAE results across 8 benchmark datasets are reported, with LVM standard deviations provided separately.The supplied passages identify the results table but do not provide its cell values.Source: §B.2 · p. 17, Table 11 · p. 18, Table 12 · p. 18

B.3 Full Results of RQ1: What type of LVM best fits TSC (TSF) task?

The comparison finds that supervised and self-supervised LVMs perform comparably on TSC, while self-supervised LVMs outperform supervised counterparts on TSF.

B.3 Full Results of RQ1: What type of LVM best fits TSC (TSF) task?

B.4 Full Results of RQ2: Which imaging method best fits TSC (TSF) task?

The imaging comparison identifies GAF as the best method for TSC and UVH as advantageous for TSF. Fine-tuning outcomes differ by task: full tuning leads on TSC, while norm-only tuning leads on TSF.

B.4 Full Results of RQ2: Which imaging method best fits TSC (TSF) task?

  • GAF† constrains predicted values to the look-back window's minimum and maximum, a limitation associated with significant performance degradation in TSF.The study uses GAF rather than GAF† in the TSC/TSF critical-difference comparison because GAF outperforms GAF† in Table 16.Source: §B.4 · p. 20, Table 16 · p. 20
  • UVH demonstrates a clear advantage on 4 TSF datasets, while GAF is the best imaging method for TSC when all datasets are used.The full comparisons cover eight imaging methods; using all TSC datasets improves confidence in identifying the best method.Source: §B.4 · p. 19, Table 16 · p. 20, Table 15 · p. 20

B.5 Full Results of RQ3: Are the pre-trained parameters in LVMs useful in time series tasks?

  • Fully fine-tuning all parameters yields the best TSC performance, whereas fine-tuning only norm layers performs better than the other TSF settings.The comparisons include full tuning, progressively frozen components, zero-shot use, and training a randomly initialized LVM from scratch.Source: §B.5 · p. 20, Table 17 · p. 21, Table 18 · p. 21

B.6 Full Results of RQ4: How useful are LVMs’ architectures?

The ablations show that pretrained LVMs outperform simplified alternatives, but LVMs are more vulnerable to temporal perturbations and trade inference efficiency for performance. Forecasting results also highlight the importance of pretrained decoders.

B.6 Full Results of RQ4: How useful are LVMs’ architectures?

  • Without pretraining, LVMs perform on par with W/O-LVM and LVM2ATTN on both tasks, but pretrained LVMs outperform both ablations.W/O-LVM replaces Transformer blocks with a linear layer; LVM2ATTN replaces them with a single multi-head self-attention layer.Source: §B.6 · p. 22, Table 19 · p. 22, Table 20 · p. 22

B.7 Full Results of RQ5: Do LVMs capture temporal order of time series?

B.8 Full Results of RQ6: What are the computational costs of LVMs?

  • LVMs can yield improved performance at higher inference-time costs on the TSC and TSF comparisons.The figures compare TSC accuracy and TSF MSE and MAE against the two best-performing baselines for each task.Source: §B.8 · p. 25, Figure 12 · p. 25, Figure 13 · p. 25

B.9 Full Results of RQ7: Which component of LVMs contributes more to forecasting

  • On TSF, excluding the pretrained encoder performs worse than excluding the pretrained decoder, highlighting the importance of pretrained decoders.Source: §B.9 · p. 26, Table 23 · p. 26

B.10 Full Results of RQ8: Will period-based imaging method induce any bias?

The segment-length analysis identifies a forecasting-period bias and finds that LVMs have limited ability to fully use long look-back windows.

B.10 Full Results of RQ8: Will period-based imaging method induce any bias?

  • MAE is generally best when UVH segment length is a period multiple (L or 2L), and Figure 15 plots performance across segment lengths.Source: Figure 15 · p. 26, §B.10 · p. 26

B.11 Full Results of RQ9: Can LVMs make effective use of look-back windows?

  • LVMs show limited ability to fully leverage look-back windows longer than approximately 1,000 time steps.Table 24 covers look-back lengths from 48 to 2,304; Illness is omitted because its series total only 966 time steps.Source: Table 24 · p. 27, §B.11 · p. 27

C Proof of Lemma 1

The proof turns segment recurrence in a perfectly periodic series into a divisibility condition, then uses the greatest common divisor to obtain the smallest recurrence count.

C Proof of Lemma 1

D.1 Visualization of GAF on TSC Task

The visualizations show class-distinguishing GAF patterns for two classification datasets and illustrate how UVH forecasts depend on segment alignment with the series period.

D.1 Visualization of GAF on TSC Task

D.2 Illustration of An Inductive Bias of LVMs during TSF

Loading 2505.24030v2…