PaperlayerView exact revision on arXiv

Source-linked AI summary

OccamVTS: Distilling Vision Models to 1% Parameters for Time Series Forecasting

Sisuo Lyu, Siru Zhong, Weilin Ruan, Qingxiang Liu, Qingsong Wen, Hui Xiong, Yuxuan Liang

arXiv:2508.01727v2cs.LGcs.CV

TL;DR

Time-series forecasting has motivated using vision models to capture temporal patterns, but those models also contain semantic features that may be irrelevant or harmful. OccamVTS distills forecasting-relevant knowledge into lightweight students, achieving state-of-the-art performance while retaining only 1% of the predictive information.

  • Problem

    Vision models can capture useful visual patterns for forecasting, but high-level semantics may add irrelevant complexity and impair accuracy.

    Source: §1 · p. 2

  • Method

    OccamVTS uses pretrained vision models as teachers and distills forecasting-relevant knowledge into lightweight students through pyramid-style alignment and selective transfer.

    Source: §1 · p. 2, §3.3 · p. 5, §3.4 · p. 5

  • Results

    OccamVTS achieves state-of-the-art performance while retaining only 1% of predictive knowledge from vision models.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    The framework offers an efficient cross-modal forecasting approach that is especially valuable in data-scarce scenarios.

    Source: §5 · p. 8

Abstract

from arXiv · show

Time series forecasting is fundamental to diverse applications, with recent approaches leverage large vision models (LVMs) to capture temporal patterns through visual representations. We reveal that while vision models enhance forecasting performance, 99% of their parameters are unnecessary for time series tasks. Through cross-modal analysis, we find that time series align with low-level textural features but not high-level semantics, which can impair forecasting accuracy. We propose OccamVTS, a knowledge distillation framework that extracts only the essential 1% of predictive information from LVMs into lightweight networks. Using pre-trained LVMs as privileged teachers, OccamVTS employs pyramid-style feature alignment combined with correlation and feature distillation to transfer beneficial patterns while filtering out semantic noise. Counterintuitively, this aggressive parameter reduction improves accuracy by eliminating overfitting to irrelevant visual features while preserving essential temporal patterns. Extensive experiments across multiple benchmark datasets demonstrate that OccamVTS consistently achieves state-of-the-art performance with only 1% of the original parameters, particularly excelling in few-shot and zero-shot scenarios.

1 Introduction

Time series forecasting must handle non-stationarity, noise, long-range dependencies, and patterns spanning multiple temporal scales. OccamVTS asks how to retain vision models’ useful inductive biases while removing redundant or harmful components, distilling predictive information into compact forecasting networks.

1 Introduction

  • Forecasting predicts future values from historical observations, while its challenges include non-stationarity, stochastic noise, long-range dependencies, and local patterns alongside global trends.Deep learning approaches model numerical sequences or frequency-domain representations to capture temporal dynamics.Source: §1 · p. 1
  • Recent approaches transform time series into image-like representations and use vision models to detect edges, gradients, and frequency textures as temporal patterns.Examples include TimesNet, VisionTS, and TimeVLM.Source: §1 · pp. 1–2
  • Time-series features align with texture-rich images but diverge from semantically complex scenes, suggesting that high-level visual semantics can introduce overfitting to irrelevant features.The analysis uses MAE features from ImageNet and four time-series benchmarks; the redundancy is particularly detrimental in limited-data scenarios.Source: §1 · p. 2
  • Two modality mismatches are identified: images favor translation invariance while time series depend on absolute position, and vision models prioritize semantics absent from numerical signals.These mismatches are linked to computational inefficiency, representational redundancy, and objectives misaligned with forecasting regression.Source: §1 · p. 2
  • OccamVTS distills the most salient 1% of predictive information from pretrained vision models into compact forecasting networks.It uses native pretrained vision models as teachers and pyramid-style feature alignment with selective distillation.Source: §1 · p. 2

2 Related Work

Prior work has transformed time series into visual forms and adapted vision backbones, but which visual features help forecasting and which add harmful noise remains an open challenge. OccamVTS selectively distills visual knowledge into a lightweight model.

2 Related Work

  • Prior work has explored image conversion with CNNs, structural methods such as TimesNet and TimeMixer++, and pretrained vision backbones including MAE, vision-language, and diffusion models.The cited examples include VisionTS, TimeVLM, and LDM4TS.Source: §2 · p. 2
  • The unresolved challenge is preserving predictive low-level textures while shedding irrelevant object-centric semantic features from vision backbones.OccamVTS addresses this by selectively distilling the crucial 1% of visual knowledge and pruning the rest.Source: §2 · pp. 2–3

3 Methodology

OccamVTS selectively transfers forecasting-relevant visual knowledge from a pretrained vision teacher to a lightweight student. Its method combines texture-oriented cross-modal representations, an asymmetric teacher–student design, and selective multi-scale distillation.

3.1 Cross-Modal Representation Module

  • The cross-modal representation module pairs patch-based temporal features with frequency- and periodicity-enhanced visual augmentations transformed into multi-scale textures.Lightweight 1D and shallow 2D convolutions create spatial patterns, which are reorganized into images for pretrained vision encoders.Source: §3.1 · pp. 3–4

3.2 Teacher-Student Model Design

  • A frozen pretrained vision backbone supplies visual features to a teacher that fuses them with temporal features, while a compact student learns forecasting and teacher-derived knowledge.The student’s reduced visual encoder is intended to focus on forecasting-relevant features and filter semantic noise.Source: §3.2 · p. 4

3.3 Knowledge Distillation Module

  • Pyramid-style feature alignment and correlation and feature distillation transfer teacher representations and temporal attention patterns to the student.Adaptive weighting balances distillation with forecasting accuracy.Source: §3.3 · p. 5

3.4 Contributions at a glance

  • The design combines texture-oriented cross-modal representations, a frozen vision teacher with temporal-query fusion, and selective distillation to retain forecasting-relevant patterns while suppressing semantic redundancy.The distillation scheme aligns temporal attention and representations through correlation and feature objectives.Source: §3.4 · p. 5

4 Experiments

Experiments evaluate OccamVTS in long-term, few-shot, zero-shot, and short-term forecasting, alongside efficiency, ablation, and scalability analyses. Across these settings, the reported results favor the distilled model, including under limited data.

4.1 Experimental Settings

  • Experiments use eight benchmarks, with MAE-Large as the default teacher and Tiny-ViT as the student, corresponding to approximately 1% of the teacher’s parameters.Long-term forecasting uses MSE and MAE; M4 short-term evaluation uses SMAPE, MASE, and OWA.Source: §4.1 · pp. 5–6

4.2 Long-term Forecasting

  • Across seven long-term benchmarks, OccamVTS outperforms state-of-the-art baselines, including 12.0% lower MSE than PatchTST on ETTh2.The authors also report improvements over TimeMixer++ on Electricity and iTransformer on Traffic.Source: Table 1 · p. 6, §4.2 · p. 6

4.3 Few-shot Forecasting

  • With 10% of training data, OccamVTS achieves the best MSE and MAE on five of seven datasets, including a 4% MSE reduction on ETTm2 versus the second-best performer.The complete method exceeds its non-distilled variant by 3.0–4.6% across datasets in this setting.Source: Table 2 · p. 6, §4.3 · p. 6

4.4 Zero-shot Forecasting

  • In zero-shot ETT transfer, OccamVTS leads in 5 of 8 MSE and 7 of 8 MAE metrics, with 13.5% and 6.7% MSE improvements over TimeVLM on two challenging transfers.The authors report only ±8% performance variation across transfer pairs, compared with 20–40% degradation for most baselines.Source: §4.4 · p. 6, Table 3 · p. 7

4.5 Short-term Forecasting

  • 1.3% SMAPE improvement over traditional approaches is reported for short-term forecasting, with the distilled method also improving 1.4–2% over its non-distilled variant across metrics.The evaluation spans forecasting horizons of 6–48 time steps.Source: Table 4 · p. 7, §4.5 · p. 7

4.6 Model Analysis

  • On ETTh1, distilled students use less than 1–2% of teacher parameters and outperform massive teacher models while reducing parameters by 99%.Figure 3 compares MAE, inference time, and parameter count across four forecasting horizons.Source: Figure 3 · p. 7, §4.6 · p. 7
  • On Weather, removing vision worsens MSE by 4.4%, while removing temporal components raises it by 36.7%; Figure 4 ablations also favor the full distillation framework.The reported component comparisons include removing correlation or feature distillation, teacher-only, and student-only variants.Source: Table 5 · p. 7, §4.6 · pp. 7–8, Figure 4 · p. 8
  • Across four datasets, the distilled model outperforms the non-distilled variant at every tested training-data level, with the gap most pronounced at 20–40% data.On ETTh1 and ETTh2, the reported MSE/MAE gap reaches 10–15% in low-data scenarios.Source: Figure 5 · p. 8, §4.6 · p. 8

5 Conclusion

OccamVTS distills predictive knowledge from vision models into compact students for time-series forecasting. The paper reports state-of-the-art performance with reduced computational overhead and highlights data-scarce settings as especially valuable.

5 Conclusion

  • OccamVTS is a knowledge-distillation framework designed to eliminate visual-model redundancy in time-series forecasting.It distills the essential 1% of predictive knowledge into lightweight student networks.Source: §5 · p. 8
  • The authors report state-of-the-art performance while dramatically reducing computational overhead through distillation.They describe the approach as preserving cross-modal benefits while eliminating harmful redundancy.Source: §5 · p. 8
  • The conclusion identifies data-scarce scenarios, where traditional methods suffer from overfitting, as a particularly valuable setting for OccamVTS.Future work may explore foundation models, multi-expert ensembles, modalities beyond vision, and domain-specific compression ratios.Source: §5 · p. 8

A Dataset Details

The appendix describes the benchmark collection, evaluation measures, model and training configurations, distillation settings, and teacher–student choices used in the experiments.

A Dataset Details

  • The benchmark suite covers forecasting domains including electricity, transportation, and meteorology, with datasets varying in frequency, dimensionality, and sequence length.The M4 benchmark adds six temporal granularities and multiple domains.Source: A Dataset Details · p. 11, Table 6 · p. 11

B Evaluation Metrics

  • Long-term forecasting is evaluated with MSE and MAE, while M4 uses SMAPE, MASE, and OWA under its competition framework.The metric formulations use series periodicity and prediction horizon.Source: B Evaluation Metrics · p. 12

1 Model Architecture Parameters

  • The default architecture uses 56 × 56 input representations, a 128-dimensional hidden size, two encoder layers, one decoder layer, and a 256-dimensional fusion layer.The configuration also specifies dropout, a memory bank, and top-k retrieval settings.Source: §1 · p. 12

2 Training Parameters

  • Training uses 70%/10%/20% train/validation/test splits, AdamW, batch size 32, and a maximum of 10 epochs.Input length is 512, with prediction horizons of 96, 192, 336, or 720 according to task.Source: §2 · pp. 12–13, Table 8 · p. 13

3 Distillation Parameters

  • Distillation uses three hierarchical feature-alignment scales and learnable loss weights, with temperature initialized to 4.The listed initial weights include 0.01 for feature distillation and 1 for the forecasting task.Source: §3 · p. 14, Table 9 · p. 14

4 Teacher and Student Model

  • The framework supports six heterogeneous pretrained vision teachers and lightweight student backbones, with all students fixed at a 128-dimensional hidden size.The evaluated teachers span feature dimensions from 512 to 2048; students include Tiny-ViT, EfficientNet-B0, and MobileNet-V3.Source: §4 · p. 14, Why these three student models? · p. 15

Why these three student models?

  • The three student models cover transformer and CNN architectures, and together yield 52–99% fewer parameters than their teachers.The paper presents these choices as a test across distinct architectural priors and deployment-friendly backbones.Source: Why these three student models? · p. 15

D Long-term Forecasting

OccamVTS achieves the best MSE across seven long-term forecasting datasets and the best or second-best MAE on them, with reported gains on ETT and high-dimensional datasets.

D Long-term Forecasting

E Few-shot Forecasting

With 10% of training data, OccamVTS remains competitive and often outperforms comparison methods; in zero-shot ETT transfers, it leads on most reported metrics.

E Few-shot Forecasting

F Zero-shot Forecasting

  • In zero-shot ETT transfers, OccamVTS achieves the best results on five of eight MSE metrics and seven of eight MAE metrics.For ETTh2→ETTh1, it reports 0.429 MSE versus TimesVLM’s 0.496; ETTm2→ETTm1 reports 0.403 versus 0.451 without distillation.Source: F Zero-shot Forecasting · p. 16

G Short-term Forecasting

OccamVTS performs competitively on short-term forecasting despite a mismatch between its feature-distillation MSE loss and the evaluation metrics, with the full framework transferring temporal patterns across horizons.

G Short-term Forecasting

H Teacher Student Long Term Forecasting and Efficiency

OccamVTS reduces vision-encoder parameters substantially, but runtime gains are smaller because the temporal forecasting module is preserved; teacher choice also affects performance.

H Teacher Student Long Term Forecasting and Efficiency

  • Tiny-ViT reduces parameters by 99.5% relative to MAE-Huge, while vision-encoder reductions range from 56% to 99.9%.Table 14 compares teacher and student computational efficiency across datasets at horizon 96.Source: H Teacher Student Long Term… · pp. 17–18, Table 14 · p. 18
  • Memory falls by 17–60% while inference speed improves 1.5–3.4×, since compression targets the vision component and preserves the temporal forecasting module.On ETTh1, Tiny-ViT’s 0.65M vision encoder still requires 1714 MiB, versus 2146 MiB for ResNet101.Source: H Teacher Student Long Term… · p. 18
  • With fixed distillation weights, teacher choice affects performance more than student architecture; MAE-based teachers generally provide the most robust improvements.Table 15 compares teacher-student combinations on long-term forecasting tasks; dataset-specific advantages indicate pairing compatibility also matters.Source: H Teacher Student Long Term… · pp. 18–19, Table 15 · p. 19

I Visualization of prediction results

Across four datasets and horizons from 96 to 720 steps, the distilled model follows the ground truth more closely than the teacher without distillation, especially at inflection points and abrupt changes.

I Visualization of prediction results

  • Across four datasets and 96-, 192-, 336-, and 720-step horizons, KD predictions follow the ground truth more closely, with smaller deviations and smoother curves than the teacher without distillation.The difference is particularly visible at inflection points and abrupt changes; both models accumulate more error at longer horizons.Source: I Visualization of prediction results · p. 19, Figure 6 · p. 19, Figure 7 · p. 20

J Visualization of Time Series and ImageNet Feature Similarity

The visual analysis finds that time-series features align with texture-rich patterns and diverge from semantically complex images. The appendix details cross-modal representations, visual augmentation, and adaptive distillation, then outlines future research directions.

J Visualization of Time Series and ImageNet Feature Similarity

1 Temporal Feature Extraction

  • The temporal feature extractor segments multivariate sequences into overlapping patches, adds sinusoidal positional encodings, and uses transformer layers to capture hierarchical temporal patterns.The appendix describes the transformer as capturing local and global dependencies.Source: §1 · pp. 21–22

2 Visual Augmentation for Time Series

  • Visual augmentation emphasizes low-level textures and gradients, while multi-scale convolutions capture patterns from local variations to global trends.The resulting visual representation is designed to encode frequency, periodicity, and local trends as textures, repetitions, and gradient changes.Source: §2 · pp. 22–23

Dynamic Loss Weight Learning

Hierarchical Projection Strategy

  • Pyramid-style feature alignment uses projections at multiple granularities and learnable scale weights to address dimensional mismatch between teacher and student representations.Attention weights guide the selection of scale combinations.Source: Hierarchical Projection Strategy · pp. 23–24

Exponential Moving Average Loss Tracking; Multi-Component Loss Integration

Dual-Optimizer Training Strategy; Performance Tracking Metrics

M Future Work

  • Future work proposes extending distillation to foundation models, multiple domain-specific vision teachers, and language, audio, and sensor modalities.These directions involve preserving zero-shot generalization, selecting relevant experts, and aligning heterogeneous feature spaces.Source: M Future Work · p. 25

Related papers

Loading 2508.01727v2…