Source-linked AI summary
Distilling Time Series Foundation Models for Efficient Forecasting
Yuqi Li, Kuiye Ding, Chuanguang Yang, Szu-Yu Chen, Yingli Tian
TL;DR
Large TSFMs forecast well but are expensive to deploy, and standard distillation does not address forecasting-specific horizon imbalance or architectural mismatch. DistilTS uses horizon-weighted objectives and factorized temporal alignment to distill compact students. Experiments show comparable forecasting performance with substantially improved efficiency.
Problem
TSFMs provide strong forecasting but their large parameter sizes make deployment costly, while conventional distillation does not directly address forecasting-specific and architectural discrepancies.
Method
DistilTS combines horizon-weighted objectives with factorized temporal alignment to balance supervision across horizons and project student embeddings into the teacher’s temporal space.
Results
DistilTS achieves forecasting performance comparable to full-sized TSFMs while delivering orders-of-magnitude improvements in efficiency across benchmarks.
Takeaways & Limitations
The framework yields lightweight student models that retain the forecasting strength of large TSFM teachers with more efficient inference.
Abstract
from arXiv · showhide
Time Series foundation models (TSFMs) deliver strong forecasting performance through large-scale pretraining, but their large parameter sizes make deployment costly. While knowledge distillation offers a natural and effective approach for model compression, techniques developed for general machine learning tasks are not directly applicable to time series forecasting due to the unique characteristics. To address this, we present DistilTS, the first distillation framework specifically designed for TSFMs. DistilTS addresses two key challenges: (1) task difficulty discrepancy, specific to forecasting, where uniform weighting makes optimization dominated by easier short-term horizons, while long-term horizons receive weaker supervision; and (2) architecture discrepancy, a general challenge in distillation, for which we design an alignment mechanism in the time series forecasting. To overcome these issues, DistilTS introduces horizon-weighted objectives to balance learning across horizons, and a temporal alignment strategy that reduces architectural mismatch, enabling compact models. Experiments on multiple benchmarks demonstrate that DistilTS achieves forecasting performance comparable to full-sized TSFMs, while reducing parameters by up to 1/150 and accelerating inference by up to 6000x. Code is available at: https://github.com/itsnotacie/DistilTS-ICASSP2026.
1. INTRODUCTION
TSFMs offer strong forecasting but are costly to deploy, while existing distillation approaches do not specifically address their forecasting and architectural challenges. DistilTS introduces horizon-weighted objectives and factorized temporal alignment to produce efficient students with comparable performance.
- 1. INTRODUCTION: TSFMs achieve strong forecasting and zero-shot performance but their billion-scale parameters and limited inference efficiency create serious deployment barriers.The motivation is to preserve TSFM generalization while enabling efficient deployment.
- 1. INTRODUCTION: DistilTS is the first distillation framework specifically designed for TSFMs, targeting forecasting-specific task difficulty and teacher–student architecture discrepancies.Existing approaches distill LLMs or conventional forecasting models rather than TSFMs.
- 1. INTRODUCTION: Horizon-weighted objectives redistribute supervision toward difficult long-term horizons, while factorized temporal alignment projects student embeddings into the teacher’s temporal representation space.These components address short-horizon optimization dominance and variate-wise versus point-wise representations.
- 1. INTRODUCTION: DistilTS achieves performance comparable to full-scale TSFMs while significantly improving efficiency on real-world benchmarks.The framework also includes an open-source implementation covering representative TSFMs.
2. METHODOLOGY
DistilTS addresses forecasting-specific optimization imbalance and teacher–student representation mismatch through horizon weighting and factorized temporal alignment. These components transfer temporal knowledge while retaining a compact variate-wise student.
- 2.1.1. Horizon-weighted knowledge distillation: Later horizons receive larger weights to counter optimization dominance by easier short-term forecasts.Uniform averaging can under-train long horizons because short-term forecasts provide stronger, more consistent gradients.
- 2.1.1. Horizon-weighted knowledge distillation: The horizon parameter acts like a distillation temperature by amplifying harder long-horizon steps under practical data and computation limits.The analogy is conceptual: unlike classification temperature, it balances optimization across forecasting horizons.
- 2.1.2. Factorized Temporal Alignment: The teacher uses point-wise states across time, whereas the efficient student compresses each variate into one embedding, creating an architectural mismatch.Variate-wise tokenization reduces token count and parameter size but removes explicit time indexing from student states.
- 2.1.2. Factorized Temporal Alignment: Factorized temporal alignment projects each student variate embedding into a latent space, applies learnable time modulation, and maps it into the teacher space.The design factorizes variable identity and temporal modulation to reconstruct point-wise signals without increasing the student’s token count.
- 2.1.3. Boundary Case Analysis: FTA can exactly recover point-wise teacher representations in a boundary case and otherwise supports soft time-wise gating through learnable time embeddings.With one-hot time embeddings, the construction selects the corresponding point embedding; learnable embeddings generalize this to soft gating.
- 2.2. Alternative KD Variants: The methodology also introduces trend-projection and frequency-and-difference knowledge-distillation baselines for comparison.These alternatives distill coarse trends or align spectral and difference features.
3. EXPERIMENT
Experiments evaluate DistilTS on five long-term forecasting benchmarks, comparing accuracy, efficiency, teacher settings, and distillation variants. DistilTS remains competitive with large TSFMs while substantially reducing computational costs.
- 3.1. Experimental Settings: Experiments use ETTh1, ETTh2, ETTm1, ETTm2, and Weather under long-term forecasting, reporting averaged MSE and MAE over five runs.Student and teacher models use aligned lookback windows, and horizons 336 and 720 are excluded because they substantially exceed the input context.
- 3.2. Main Results: DistilTS ranks first in 13 benchmark cases and second in several others across five datasets and prediction lengths 96 and 192.The reported results show comparable performance to large TSFMs.
- 3.2. Main Results: DistilTS achieves the lowest error while reducing parameter scale by two orders of magnitude and accelerating inference by over three orders of magnitude.Efficiency measurements use ETTh1, while forecasting performance is averaged across five benchmarks.
- 3.3. Ablation Experiments: Teacher architecture significantly affects distillation performance, with TimeMoe and Chronos outperforming Moirai in the reported ablation.Within TimeMoe, increasing teacher size from 50M to 200M provides only limited additional benefit.
- 3.3. Ablation Experiments: DistilTS consistently improves its DLinear and iTransformer student baselines and outperforms FD-KD and KD on ETTh2 for both MSE and MAE.The ablations use the same teacher for the DistilTSL and DistilTST variants.
4. CONCLUSION
DistilTS distills time-series foundation models by addressing horizon imbalance and variate-temporal mismatch. It retains forecasting performance comparable to full-sized TSFMs while improving efficiency by orders of magnitude.
- 4. CONCLUSION: DistilTS enables lightweight student models to retain the forecasting strength of large teachers while delivering orders-of-magnitude efficiency improvements.The conclusion attributes this to addressing horizon imbalance and variate-temporal mismatch.