PaperlayerView exact revision on arXiv

Source-linked AI summary

VisionTS: Visual Masked Autoencoders Are Free-Lunch Zero-Shot Time Series Forecasters

Mouxiang Chen, Lefei Shen, Zhuo Li, Xiaoyun Joy Wang, Jianling Sun, Chenghao Liu

arXiv:2408.17253v4cs.CVcs.AIcs.LG

TL;DR

Time-series foundation models face transfer challenges associated with existing text-based approaches and heterogeneous time-series data. VisionTS repurposes an ImageNet-pretrained visual MAE by reformulating forecasting as image reconstruction. The authors report strong zero-shot performance and state-of-the-art results on most long-term benchmarks after one fine-tuning epoch.

  • Problem

    The paper asks whether a visual model pretrained on images can provide another route to TSF foundation models amid challenges associated with existing approaches and heterogeneous time-series data.

    Source: §1 · p. 2

  • Method

    VisionTS converts time-series forecasting into patch-level image reconstruction for a visual MAE pretrained on ImageNet.

    Source: §1 · pp. 2–3

  • Results

    Without time-series adaptation, vanilla MAE matches or outperforms strong zero-shot TSF models, while one fine-tuning epoch achieves state-of-the-art performance on most long-term benchmarks.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The authors suggest that visual models may offer a useful route for TSF and motivate further cross-modality research.

    Source: §1 · p. 2, §6 · p. 9

    VISIONTS cannot capture multivariate interactions or perform distribution forecasting because of limitations in the visual model.

    Source: §6 · p. 9

Abstract

from arXiv · show

Foundation models have emerged as a promising approach in time series forecasting (TSF). Existing approaches either repurpose large language models (LLMs) or build large-scale time series datasets to develop TSF foundation models for universal forecasting. However, these methods face challenges due to the severe cross-domain gap or in-domain heterogeneity. This paper explores a new road to building a TSF foundation model from rich, high-quality natural images. Our key insight is that a visual masked autoencoder, pre-trained on the ImageNet dataset, can naturally be a numeric series forecaster. By reformulating TSF as an image reconstruction task, we bridge the gap between image pre-training and TSF downstream tasks. Surprisingly, without further adaptation in the time series domain, the proposed VisionTS could achieve better zero-shot forecast performance than existing TSF foundation models. With fine-tuning for one epoch, VisionTS could further improve the forecasting and achieve state-of-the-art performance in most cases. Extensive experiments reveal intrinsic similarities between images and real-world time series, suggesting that visual models may offer a "free lunch" for TSF and highlight the potential for future cross-modality research. Our code is publicly available at https://github.com/Keytoyze/VisionTS.

1. Introduction

The paper asks whether visual models pretrained on images can provide another route to time-series forecasting, given cross-domain transfer challenges and time-series heterogeneity. It introduces VisionTS, which reformulates forecasting as image reconstruction, and reports strong zero-shot results with further gains from one-epoch fine-tuning.

1. Introduction

  • Existing TSF foundation-model approaches repurpose language models or train on time-series datasets, whose heterogeneity in length, frequency, variates, domains, and semantics limits transfer.The paper also notes that constructing high-quality time-series datasets remains challenging and is still at an early exploration stage.Source: §1 · p. 2
  • The paper evaluates VisionTS across 8 long-term TSF, 29 Monash, and 23 GIFT-Eval datasets spanning diverse domains, frequencies, and multivariate series.The authors describe this evaluation benchmark as the largest among existing TSF foundation models.Source: §1 · p. 2
  • Without time-series adaptation, vanilla MAE matches or outperforms strong zero-shot TSF models; one fine-tuning epoch achieves state-of-the-art performance on most long-term benchmarks.The evaluation also reports comparable or superior results to zero-shot TS-based models and performance surpassing few-shot text-based models.Source: §1 · p. 3
  • The authors identify similarities between time-series and image representations, alongside cross-domain heterogeneity within time-series data, as possible explanations for VisionTS transferability.They present visual models as a promising direction for further cross-modal research.Source: §1 · pp. 2–3
  • VisionTS uses a visual masked autoencoder pretrained on ImageNet and reformulates time-series forecasting as patch-level image reconstruction.The input series is segmented, rendered as an image, and aligned so the forecasting window corresponds to masked patches.Source: §1 · pp. 2–3

2. Preliminaries

The preliminaries define forecasting as predicting a future window from a historical sequence and describe MAE’s patch-reconstruction pretraining task.

2. Preliminaries

  • Given a historical sequence of context length L, TSF predicts future values over a horizon of length H.For a multivariate series with M variables, the input and prediction windows have shapes L×M and H×M, respectively.Source: §2 · p. 3
  • MAE divides a square image into an N×N grid of patches, each with width and height S = W/N.The described pretraining uses images of size W×W.Source: §2 · p. 3
  • During pretraining, MAE feeds visible patches and their position encodings to a ViT, which reconstructs the masked patches’ pixel values.Randomly selected patches are masked.Source: §2 · p. 3

3. Methodology

VisionTS converts a time series into an image whose visible patches represent the look-back window and whose masked patches represent the forecast. It uses a visual MAE to reconstruct those patches, then reverses the image transformations to obtain predictions.

3. Methodology

  • VisionTS maps the look-back window to visible patches and the forecast window to masked patches, turning TSF into MAE’s image-reconstruction task without further training.The paper relates this mapping to prompt-based prediction from masked tokens in pretrained language models.Source: §3 · p. 3, Figure 3 · p. 4
  • Segmentation reshapes a univariate series into ⌊L/P⌋ subsequences of length P, with P=1 available when no clear periodicity exists.Stacking these subsequences forms a 2D matrix that captures within-period and same-phase across-period variation.Source: §3 · p. 3
  • The method instance-normalizes the matrix to standard deviation r; the authors report r=0.4 performs well across most scenarios.They explain that reducing magnitude helps keep inputs and outputs within MAE’s limited color-value range, while too-low values can be hard to distinguish.Source: §3 · p. 4
  • VisionTS renders the normalized matrix as identical-channel grayscale, then resamples it to align the visible and masked patches with MAE’s pretrained image dimensions.The visible-patch count is based on L/(L+H); the authors also report that using c=0.4 to reduce visible width performs well, possibly by better matching MAE’s masking ratio.Source: §3 · p. 4
  • After MAE reconstructs the image, VisionTS resizes it back to the series segmentations, averages its channels, de-normalizes, and flattens it to extract the forecast.This reverses the earlier image-processing steps.Source: §3 · p. 4, Figure 3 · p. 4
  • VisionTS uses channel independence because visual models struggle to capture inter-variable interactions with limited image channels; capturing those interactions is left for future work.This means forecasting each variable separately in the approach described.Source: §3 · p. 5

4. Experiments

VISIONTS is evaluated across zero-shot and fine-tuned forecasting settings, with analyses of its backbone, computational cost, hyperparameters, modality relationships, and design choices. It performs strongly across benchmarks, including with only one epoch of fine-tuning.

4.1. Zero-Shot Time Series Forecasting

  • Zero-shot evaluation covers long-term TSF, Monash, and GIFT-Eval benchmarks; six long-term datasets were chosen to avoid overlap with MOIRAI's pre-training data.The Monash benchmark is more challenging for VISIONTS because it was included in MOIRAI pre-training but not VISIONTS; hyperparameters are selected using validation sets.Source: §4.1 · p. 5
  • VISIONTS achieves the best forecasting performance in 7 of 14 long-term cases, with about 6% lower average MSE than MOIRAISmall and MOIRAILarge.Its relative average MSE reduction versus few-shot baselines ranges from 8% to 84%, and it performs comparably to MOIRAIBase.Source: §4.1 · p. 6
  • VISIONTS ranks first by normalized MASE on GIFT-Eval and second on Monash, behind MOIRAI; on Monash it surpasses individually trained models and LLMTime.The authors note that some concurrent GIFT-Eval leaderboard works may have data leakage, whereas MAE was trained on ImageNet before the leaderboard's release.Source: §4.1 · p. 6

4.2. Further Analysis of VISIONTS

  • Three MAE variants outperform MOIRAISmall and MOIRAILarge overall, while larger variants show a slight performance decrease; LaMa performs similarly to MOIRAI.VISIONTS has nearly constant computation time as context length grows, unlike Transformer baselines with O(L^2) context-length scaling.Source: §4.2 · p. 6
  • Performance typically improves with longer context, especially on high-frequency datasets, while normalization and alignment constants near 0.4 work well across most benchmarks.MAE encoder visualizations place ETTm1 and Electricity within the ImageNet distribution, while time-series distributions are generally more scattered.Source: §4.2 · pp. 6–7
  • Bilinear interpolation performs best, and image flips do not significantly change performance; regular inputs support forecasts of periodicity and trends, while less-structured inputs can yield more volatile forecasts.The more aggressive forecasts sometimes improve trend predictions but can also produce greater MAE.Source: §4.2 · pp. 7–8

4.3. Full-Shot Long-Term Time Series Forecasting

  • VISIONTS outperforms other baselines in 46 of 80 full-shot cases, using one fine-tuning epoch on each dataset except Illness.Fine-tuning only layer-normalization layers is reported as the best strategy; fine-tuning MLP or attention layers causes significant performance drops.Source: Table 4 · p. 7, §4.3 · p. 8

5. Related Work

Prior TSF foundation-model work repurposes language models or trains on time-series data, while earlier image-based studies leave natural-image transfer to forecasting insufficiently examined. The paper situates its approach against these strands.

Text-based and Time Series-Based TSF Foundation Models

  • Text-based TSF work repurposes pretrained language models or converts series into text, while large-scale TS-based models train on data from diverse domains.The section notes concerns about modality gaps for language models and heterogeneity challenges for time-series datasets.Source: §5 · pp. 8–9

Image-Based Time-Series Analysis

  • Earlier image-based studies use images for time-series classification or forecasting, but do not deeply examine transfer from natural images to TSF.A prior image-pretrained BEiT approach fell short of leading text-based and TS-based TSF foundation models.Source: §5 · p. 9
  • The paper positions its contribution as showing that an image-based foundation model can match or surpass other TSF foundation models without further time-series adaptation.The authors identify this as, to their knowledge, the first such demonstration.Source: §5 · p. 9

6. Conclusion

VISIONTS uses natural-image pretraining to forecast time series by recasting forecasting as image reconstruction, and the paper reports strong zero-shot and full-shot results. The authors identify limits in multivariate interactions and distribution forecasting.

6. Conclusion

  • VISIONTS reformulates time-series forecasting as image reconstruction using a visual masked autoencoder pretrained on natural images.The paper presents this as an alternative to traditional text-based and time-series-based approaches.Source: §6 · p. 9
  • Extensive evaluations show strong forecasting performance in both zero-shot and full-shot settings.The authors describe the approach as a potential source of future cross-modality research.Source: §6 · p. 9
  • VISIONTS cannot capture multivariate interactions or perform distribution forecasting because of limitations in the visual model.The authors point to future model-structure modifications as a possible direction for adding time-series capabilities.Source: §6 · p. 9

A.1. Benchmark and baselines

This block documents benchmark coverage, baseline comparisons, periodicity selection, and hyperparameter choices for long-term forecasting. Reported comparisons show VISIONTS leading many long-term results and performing strongly against alternative forecasting models.

A.1. Benchmark and baselines

  • Evaluation covers 8 long-term datasets, 29 Monash datasets, and 23 GIFT-Eval datasets, with representative TS-based, text-based, and conventional forecasting baselines.Long-term results use MSE and MAE; Monash uses MAE, and GIFT-Eval uses MAPE.Source: §A.1 · p. 13

A.2. Periodicity selection

  • Selecting period P from a frequency-based range on the validation set significantly outperforms the naive setting P = 1.The candidate range and final dataset-specific periods are listed in Tables 5 and 6.Source: Table 7 · p. 14, §A.2 · p. 15

B.1. Hyperparameters

  • The zero-shot long-term setup tunes context length L on validation data and reports the final hyperparameters in Table 8.The supplied passages specify tuning for L but do not give the selected values.Source: Table 8 · p. 15, §B.1 · p. 15

B.2. Full forecasting results of the long-term TSF benchmark

  • VISIONTS achieves the best long-term results in 32 out of 62 cases, versus MOIRAIBase’s 10 and MOIRAILarge’s 8.The full table reports zero-shot and few-shot results.Source: §B.2 · p. 15

B.3. Comparison of TimesFM and LLMTime

  • VISIONTS outperforms TimesFM and LLMTime in MAE on the last test window of the original split.The comparison uses the same setting as the reported TimesFM and LLMTime results.Source: §B.3 · p. 15

B.4. Comparison of traditional methods

  • On the four ETT datasets, VISIONTS achieves the best performance against traditional methods, including ARIMA, ETS, Seasonal Naïve, and Seasonal Avg.ARIMA and ETS are compared only on these small-scale benchmarks because of their high computational cost.Source: §B.4 · p. 15

B.5. Comparison of concurrent works

  • Comparisons with Time-MoE, TTM, and CALF highlight the potential of vision models for time-series forecasting.The tables cover zero-shot comparisons for Time-MoE and TTM, and zero-shot and full-shot comparisons for CALF.Source: §B.5 · p. 15

B.6. Full forecasting results of the Monash TSF benchmark

The Monash results report normalized MAE across datasets and examine backbone size, image encoding choices, and hyperparameter sensitivity. Findings include effectiveness on some datasets without significant periodicity and similar performance across MAE sizes.

B.6. Full forecasting results of the Monash TSF benchmark

  • VISIONTS outperforms MOIRAI on some Monash datasets with P = 1, including FRED-MD and NN5 Weekly.Table 15 reports MAE test results, normalizing each dataset’s MAE by the naive forecast’s MAE and aggregating with the geometric mean.Source: §B.6 · p. 16, Table 15 · p. 19

B.7. Impact of backbones

  • The 112M, 330M, and 657M MAE variants perform similarly, with a slight decrease for larger models; the smallest excels on four datasets and the largest on Electricity.Table 16 also compares VISIONTS with LaMa as an alternative visual backbone.Source: §B.7 · p. 16, Table 16 · p. 19, Table 17 · p. 20

B.8. Impact of the different image encoding strategies

  • Bilinear and Bicubic interpolation perform similarly and significantly outperform Nearest Neighbor, while image orientation has little impact.The comparison assesses resampling filters and image orientations in the alignment step.Source: §B.8 · p. 16, Table 18 · p. 21

B.9. Hyperparameter analysis

  • The hyperparameter analysis plots average MSE across prediction lengths 96, 192, 336, and 720 against context length L, normalization constant r, and alignment constant c.The supplied passages describe the plotted variables and axes but do not state the direction of the trends.Source: §B.9 · p. 17, Figure 10 · p. 22, Figure 8 · p. 22

C.1. Training details

Full-shot fine-tuning treats each series variable as an individual sample and uses Adam with fixed optimizer settings. Training is one epoch for most datasets, with a longer early-stopped run for Illness.

C.1. Training details

  • Fine-tuning uses channel independence, Adam at learning rate 0.0001, and batch size 256; training runs one epoch except Illness, which runs up to 100 epochs with early stopping.Experiments are repeated three times, and r, c, and L are tuned on validation sets; final settings are listed in Table 19.Source: §C.1 · p. 23, Table 19 · p. 23

C.2. Full results and standard deviations

The full-shot results are accompanied by standard deviations computed from three runs, with text-based model results included for reference.

C.2. Full results and standard deviations

  • Full-shot results include standard deviations computed over three runs, alongside Time-LLM and GPT4TS results for reference.The full results and standard deviations are presented in Tables 21 and 20, respectively.Source: Table 20 · p. 23, §C.2 · p. 23

C.3. Ablation study and fine-tuning strategy comparison

The ablations indicate that visual knowledge is crucial to VisionTS, and that fine-tuning layer normalization is the best strategy. The section compares model variants and fine-tuning choices, with results averaged across four prediction lengths.

C.3. Ablation study and fine-tuning strategy comparison

  • Model ablations compare removing transformer blocks with replacing them by randomly initialized attention or Transformer blocks, or randomly initializing the full vanilla-MAE architecture.The variants are w/o VM, VM2Attn, VM2Trsf, and Rand-VM, respectively.Source: §C.3 · pp. 23–24
  • Fine-tuning comparisons include all weights, layer normalization, linear-layer biases, feed-forward layers, self-attention layers, and no weights.Layer normalization is the default; the frozen variant uses a shorter context than the earlier zero-shot experiment.Source: §C.3 · pp. 24–25
  • The ablations identify visual knowledge as crucial to VisionTS, and layer-normalization fine-tuning as the best strategy.Table 22 reports ablation and fine-tuning results averaged over prediction lengths 96, 192, 336, and 720.Source: §C.3 · p. 25, Table 22 · p. 25

D. Visualization

The visualization presents zero-shot VisionTS forecasts on ETTh1 and ETTh2, including examples with regular and less obvious patterns. It also shows an ETTh1 case where VisionTS underperforms MOIRAILarge.

D. Visualization

  • Figures 11–13 show zero-shot examples where VisionTS performs well, spanning a regular pattern and less obvious patterns.The figures show input/output images alongside forecasting visualizations for ETTh1 and ETTh2.Source: D. Visualization · p. 25, Figure 11 · p. 26, Figure 12 · p. 27
  • In that case, VisionTS aggressively predicts a trend despite unclear input patterns, while MOIRAILarge predicts more conservatively.Seasonal Naïve predictions and MAE metrics are also included for comparison.Source: D. Visualization · p. 25, Figure 14 · p. 29
  • Figure 14 shows an ETTh1 case where VisionTS underperforms MOIRAILarge in MAE.The figure includes input/output images and forecasting visualizations.Source: D. Visualization · p. 25, Figure 14 · p. 29
Loading 2408.17253v4…