PaperlayerView exact revision on arXiv

Source-linked AI summary

Loss-Guided Pretraining Data Selection for Time-Series Foundation Models

Yike Li, Shaoxu Song, Jianmin Wang

arXiv:2609.37255v1cs.LGcs.AI

TL;DR

TSFM pretraining draws on heterogeneous data, but its windows are often sampled without estimating their learning signal, and windows can be redundant or unlearnable. The paper scores windows with a frozen reference forecaster and retains an intermediate loss interval within each source; it reports better results than random and full-data pretraining.

  • Problem

    TSFM windows are commonly sampled without estimating their learning signal, although windows can be redundant or contain changes with no stable learnable relation.

    Source: §1 · p. 1

  • Method

    A frozen reference forecaster assigns window-loss difficulty scores, and source-stratified selection retains the central α fraction of each source.

    Source: §1 · p. 2

  • Results

    The method outperforms random selection and even surpasses full-sample pretraining across architecturally distinct TSFMs.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The central interval removes low-loss windows with limited residual learning signal and avoids overconcentrating on the unstable high-loss tail while preserving source coverage.

    Source: §5 · p. 10

Abstract

from arXiv · show

Time series foundation models (TSFMs) are pretrained on heterogeneous collections containing billions of observations, yet their training windows are typically sampled without estimating whether they provide useful learning signal. We introduce a static data-selection framework that scores each window with a reference forecaster and retains an intermediate interval within every source dataset. Specifically, we connect forecasting loss to optimization difficulty by showing that normalized squared loss controls the per-sample gradient norm under a local Jacobian condition. We then define a reference loss score and apply dataset-stratified selection to preserve the diversity of samples. Across various TSFM architectures, our method outperforms random selection by an absolute margin and even improves both relative MASE and CRPS over full-data pretraining by retaining fewer candidate pretraining windows. Further analyses show strong cross-scale and cross-architecture score correlations, indicating that a small reference model can often select data for larger targets, provided that the reference and target share compatible difficulty orderings.

1 Introduction

TSFMs rely on heterogeneous pretraining corpora, but equal attention to every window is costly and may waste learning capacity on redundant or unlearnable patterns. The paper proposes frozen-reference-loss selection with within-source retention, connecting loss to update magnitude and reporting gains over random and full-data pretraining.

1 Introduction

  • As TSFMs scale over heterogeneous corpora, treating every available window equally becomes increasingly costly and scientifically questionable.These models transfer temporal structure to unseen forecasting tasks.Source: §1 · p. 1
  • Useful windows lie between redundant easy patterns and high-loss abrupt changes that offer no stable relation for a forecaster to learn.The central challenge is finding a subset from a multi-source corpus that can match or exceed full-data training.Source: §1 · p. 1
  • A frozen reference forecaster scores each window, and within-source percentile ranks retain the central α fraction using one offline pass and one scalar per window.The approach avoids gradient computation and does not modify target training online.Source: §1 · p. 2
  • The paper presents a first systematic study of loss-based window selection for TSFM pretraining from scratch, selecting windows once before target optimization.The setting is static rather than online during target training.Source: §1 · p. 2
  • Normalized MSE controls per-sample gradient magnitude under a local-Jacobian condition, motivating reference loss as a window-difficulty score.The paper also notes that actual one-step utility depends on gradient alignment.Source: §1 · p. 2
  • The proposed method outperforms random selection and even surpasses full-sample pretraining across architecturally distinct TSFMs.The introduction reports these comparisons and says further studies examine score transfer and selection components.Source: §1 · p. 2

2 Related Work

Prior work spans universal TSFM architectures, data-selection methods in language-model pretraining, and time-series selection focused mainly on task-specific training or fine-tuning. This paper places window-level, loss-based selection in the TSFM pretraining-from-scratch setting.

2.1 Time-series foundation models

  • TSFMs vary in how they represent observations and forecasts, while this study focuses on which windows from heterogeneous corpora should be used for pretraining.Examples include TimesFM's patched continuous inputs, Chronos's scaled and quantized observations, and Moirai's masked-encoder approach.Source: §2.1 · p. 3

2.2 Data selection for large language models

  • LLM data selection ranks or filters documents by quality, redundancy, reference-model loss, or domain-level estimates, whereas this study scores time-series windows by reference-model loss.The passage also describes online methods that adapt sampling probabilities during training.Source: §2.2 · p. 3

2.3 Data selection for time series

  • Existing time-series selection chiefly addresses valuation, task-specific training, or fine-tuning; the paper studies loss-based sample selection for TSFM pretraining from scratch.Figure 2 summarizes collecting source windows, assigning frozen-reference scores, ranking within each source, and retaining a central interval.Source: §2.3 · p. 3, Figure 2 · p. 4

3 Method

The method selects forecasting windows before target training by scoring them with a frozen reference forecaster and retaining an intermediate range within each source dataset. It motivates the score through a connection between loss and gradient magnitude, then uses within-source ranks to preserve dataset diversity.

3.1 Problem formulation

  • The goal is to retain a subset of candidate windows that preserves or improves zero-shot performance while reducing the number of windows stored and processed.Each sample contains a context, future window, and mask marking observed future positions.Source: §3.1 · p. 4

3.2 Reference loss

  • Normalized MSE provides an optimization-difficulty signal because it controls per-window gradient magnitude under a local Jacobian condition.When window sizes and local Jacobian conditioning are comparable, loss rankings approximately rank update magnitudes; however, useful updates also need to align with the candidate-pool gradient.Source: §3.2 · p. 5

3.3 Dataset-stratified selection

  • Within each source, the method ranks windows by RFL percentile and retains the central α fraction, filtering by relative difficulty while preserving source coverage and dataset-level diversity.The intermediate range balances low-RFL windows expected to provide limited update magnitude against high-RFL windows that may mix useful patterns, shifts, and unlearnable noise.Source: §3.3 · p. 6

3.4 Algorithm and computational cost

  • The offline algorithm scores each candidate once, sorts scores separately within each source, and reuses them across target sizes and architectures.Scoring requires one reference-model forward pass per candidate window; sorting costs O(n_k log n_k) within source k, and one scalar is stored per window.Source: §3.4 · p. 6

4 Experiments

The experiments test whether dataset-stratified reference-loss selection improves pretraining, transfers across reference models, and benefits from its intermediate-RFL and stratification choices. Across three target architectures, the reported results favor selection over random subsets and, at 80% retention, over full-data pretraining.

4.1 Experimental setup

  • Pretraining data spans more than 100 source datasets and 4.64M univariate series, while evaluation uses 91 GIFT-Eval task configurations, with MASE and CRPS normalized by seasonal-naive forecasts and geometrically averaged across tasks.Source: §4.1 · p. 7

4.2 Overall Comparison

  • At 80% retention, dataset-stratified RFL selection improves over both full-data pretraining and size-matched random subsets on all three target models.The comparison uses the same computational budget; Table 1 reports GIFT-Eval results at different retention ratios.Source: §4.2 · p. 8, Table 1 · p. 8

4.3 Reference-model analysis

  • Within-architecture reference models are largely robust to scale, with pairwise RFL rank correlations of 0.977–0.986 for Chronos-Bolt and 0.991 for Moirai.All tested within-architecture subsets improve MASE and CRPS over full-data pretraining; a smaller reference can often approximate a larger model’s window ranking.Source: §4.3 · p. 8, Table 2 · p. 9
  • Cross-architecture transfer is conditional: TimesFM-selected data helps Chronos-Bolt but performs poorly for Moirai, where score agreement is weak.The reported rank correlations are 0.828 between TimesFM and Chronos-Bolt Mini, versus 0.254 between TimesFM and Moirai-Small.Source: §4.3 · p. 8

4.4.1 RFL region

  • Central RFL selection outperforms the low- and high-loss tails by retaining learnable windows while avoiding redundant easy cases and unstable, potentially unlearnable cases.Figure 3(a,b) shows Central among the lowest validation-loss trajectories, especially when the retained budget is small.Source: §4.4.1 · p. 9, Table 3 · p. 10

4.4.2 Dataset stratification selection

  • Dataset-stratified selection improves GIFT-Eval metrics over global selection at all tested retention ratios, preserving source coverage and diversity.Figure 3(c) compares the two selection rules; the advantage persists from 40% to 80% retention.Source: §4.4.2 · pp. 9–10, Figure 3 · p. 10

5 Conclusion

The conclusion presents dataset-stratified reference-loss selection as a way to choose intermediate-difficulty training windows while retaining source diversity. Across three TSFM families, it reports gains over size-matched random selection and full-sample training.

5 Conclusion

  • Across three distinct TSFM families, the method outperforms size-matched random selection and full-sample training.The reported framework retains an intermediate reference-loss interval within every source dataset.Source: §5 · p. 10
  • The method removes low-loss windows with limited residual learning signal and avoids overconcentrating on the unstable high-loss tail while preserving source coverage.Selection is performed independently within each source dataset.Source: §5 · p. 10
  • The paper concludes that pretraining-data selection is an important design axis alongside architecture and corpus scale for TSFMs.Source: §5 · p. 10

A Proof of Theorem 3.1

The proof uses residuals at observed future positions to derive squared-norm and matrix identities, then substitutes the resulting identity to establish Equation 6.

A Proof of Theorem 3.1

  • The residual vector contains the M_i residuals at observed future positions.Source: A Proof of Theorem 3.1 · p. 13
  • The proof takes the squared Euclidean norm of the residual vector and establishes that the stated matrix is positive semidefinite.The passage gives the quadratic form u⊤A_i u = ∥J⊤u∥².Source: A Proof of Theorem 3.1 · p. 13
  • Substituting the identity into Equation 14 and dividing by M² yields Equation 6.Source: A Proof of Theorem 3.1 · p. 13

B Additional reference-model analysis

Reference-loss rankings are stable across scales within an architecture, but cross-architecture reuse hinges on compatible difficulty orderings. TimesFM ranks Chronos candidates compatibly yet transfers poorly to Moirai.

B Additional reference-model analysis

  • Moirai-scale reference models have a Spearman correlation of 0.991, while Chronos-Bolt Mini, Small, and Base also show high rank agreement.The agreement suggests relative window difficulty is stable across reference scales within an architecture.Source: B Additional reference-model analysis · p. 13, Figure 4 · p. 14
  • ρ = 0.828 for TimesFM versus Chronos-Bolt Mini on the Chronos-Bolt-Base pool, supporting useful cross-architecture subset selection.Dataset-stratified selection depends on ranks rather than absolute loss magnitudes.Source: B Additional reference-model analysis · p. 13, Figure 4 · p. 14
  • ρ = 0.254 for TimesFM versus Moirai-Small on the Moirai pool, and the TimesFM-selected subset underperforms both the Moirai-Small subset and full-data pretraining.The authors suggest differing error objectives may lead the references to rank high-uncertainty windows differently.Source: B Additional reference-model analysis · p. 13, Figure 4 · p. 14

C Qualitative analysis

The qualitative analysis uses global RFL bands to illustrate what low, middle, and high scores represent. Middle-score windows retain learnable structure, while the tails tend to include redundant or weakly learnable examples.

C Qualitative analysis

  • The qualitative display uses global RFL bands: the bottom 1%, 49.5–50.5%, and top 1%; main experiments instead select within each dataset.The global bands are used only to make score semantics visually interpretable.Source: C Qualitative analysis · p. 14, Figure 5 · p. 15
  • Middle-score windows retain learnable trend, seasonality, or amplitude changes, while low-score windows are often regular and high-score windows often contain abrupt shifts or context–future mismatch.Low-score windows can be redundant, whereas the high-score tail mixes rare events with weakly learnable windows; this motivates removing both extremes for a compact subset.Source: C Qualitative analysis · p. 14, Figure 5 · p. 15
  • Blue marks context and red marks forecast horizon; panels show standardized windows and each pct label gives the window’s global empirical RFL percentile.The windows are standardized using the valid context mean and standard deviation.Source: Figure 5 · p. 15
Loading 2609.37255v1…