PaperlayerView exact revision on arXiv

Source-linked AI summary

Distillation of Synthetic Data for Time Series Foundation Models

Niloy Biswas, Noureddine El Karoui

arXiv:2609.09586v1stat.MLcs.LG

TL;DR

Synthetic-data pre-training for TSFMs commonly compares predictions with one realized future trajectory, creating an opportunity to use the known conditional forecast distribution instead. SDD performs this distillation, preserving expected gradients while reducing gradient covariance; on Gaussian Process data, it reached the same validation loss with 38–46% fewer FLOPs, while broader scope and production-scale persistence remain future work.

  • Problem

    Real-world time series are scarce, while Status Quo training uses only one realized future trajectory despite known conditional distributions for many synthetic generators.

  • Method

    SDD replaces realized-future losses with analytically tractable losses based on the conditional forecast distribution, integrating out future-sampling noise.

  • Results

    38–46% fewer FLOPs achieved the same validation loss as Status Quo on Gaussian Process data, with convergence speedups of 1.6× to 1.85×.

  • Takeaways & Limitations

    SDD provides unbiased, lower-covariance gradients and faster TSFM pre-training when the synthetic generator’s conditional quantities are tractable.

  • Takeaways & Limitations

    The paper leaves open whether compute savings persist at production TSFM scale and data mixtures, where synthetic data is only part of the corpus.

Abstract

from arXiv · show

Time series foundation models (TSFMs) are increasingly pre-trained on synthetically generated time series trajectories, where the data generating process is known. Current pre-training recipes are based on loss objectives which compare TSFM outputs to realized future values of each trajectory. We instead propose loss objectives which compare TSFM outputs to the conditional forecast distribution of each trajectory, a procedure we call synthetic data distillation (SDD). SDD corresponds to a Rao-Blackwellization of the training objective, in that it leaves the expectation of stochastic gradients unchanged while provably reducing the covariance of the stochastic gradient under the Loewner partial ordering. We empirically validate SDD on a TSFM model family of sizes from $4$M to $2.5$B parameters, and observe faster convergence of validation loss at every model size: on Gaussian Process data, SDD attains or improves upon the Status Quo loss whilst requiring $10\%-40\%$ less training iterations.

1 Synthetic data for Time Series Foundation Models pre-training

TSFMs are pre-trained on diverse time-series data for forecasting, but scarce public real-world series have made synthetic data central to their training. The paper introduces SDD to use synthetic trajectories more efficiently.

  • TSFMs are large neural networks pre-trained across domains to forecast unseen time series, enabling zero-shot forecasting.
  • Publicly available real-world time series are scarce, motivating synthetic data generation for TSFM pre-training.
  • SDD condenses data from infinitely many synthetic trajectories into a single loss calculation without generating the trajectories.
  • SDD accelerates TSFM pre-training by reducing stochastic-gradient variance during training.

2 Distillation of Synthetic Data for Time Series Foundation Models

SDD replaces realized future targets with analytically tractable conditional forecast information from synthetic generators. It preserves expected gradients while reducing their covariance, with tractability-dependent implementation options and faster empirical convergence.

  • 2.2 Synthetic Data Distillation uses the exact conditional distribution of future trajectories: Status Quo trains against one realized future trajectory, whereas SDD uses the known conditional distribution of future values given history.
  • 2.2 Synthetic Data Distillation uses the exact conditional distribution of future trajectories: For point forecasts with MSE, SDD uses the conditional future mean as the target, up to a θ-independent additive constant.
  • 2.3 General methodology and implementation considerations: SDD replaces the ordinary loss with a distilled loss having the same expectation while integrating out sampling noise from realized futures.
  • 2.3 General methodology and implementation considerations: SDD requires tractable conditional means, variances, or CDFs; applicable generators include Gaussian processes, linear Gaussian state-space models, and several stochastic processes.
  • 2.3 General methodology and implementation considerations: For intractable generators, Monte Carlo estimates or fallback to Status Quo can retain SDD for tractable portions of the corpus.
  • 2.4 Variance Reduction Guarantees: SDD gradients are unbiased and have covariance no greater than Status Quo gradients in the Loewner ordering.
  • 2.4 Variance Reduction Guarantees: This variance reduction yields faster convergence during TSFM pre-training.

3 Numerical Experiments

The experiments evaluate SDD on five Toto-2 architectures from 4M to 2.5B parameters using Gaussian-process synthetic data. Across model sizes, SDD reaches Status Quo validation loss with substantially lower compute and performs better at matched compute.

  • Experimental setup: Five Toto-2 architectures spanning 4M to 2.5B parameters were pretrained from random initialization on univariate Gaussian-process trajectories.The trajectories had length T = 512, with analytically tractable conditional forecast distributions.
  • Experimental setup: Held-out next-patch CRPS was measured after 300,000 training steps to track Status Quo and SDD validation loss.Training used contiguous patch masking, nine-decile quantile heads, batch size B = 16, and a single-pass regime.
  • Results: 38-46% fewer FLOPs were required for SDD to attain the same validation loss as Status Quo, corresponding to a 1.6× to 1.85× convergence speed-up.The conditional-distribution moments used by SDD were cached beforehand, so SDD and Status Quo steps had equal training cost.
  • Results: SDD consistently attained lower validation loss than Status Quo at the same training compute.Figure 2 compares held-out next-token CRPS against cumulative training compute estimated as 6ND.
  • Results: At 1B parameters, SDD reached the 300,000-step Status Quo loss in 185,000 steps, a 1.62× speed-up and 38% compute saving.The figure defines speed-up as 300,000 divided by the SDD iterations needed to reach the Status Quo endpoint loss.

4 Discussion and Future Work

The paper presents SDD as a way to accelerate TSFM pre-training with synthetic data. It identifies broader evaluations needed to establish how far the reported compute savings generalize.

  • Discussion: SDD is introduced as a method for accelerating the pre-training of time series foundation models with synthetic data.
  • Future Work: Future work includes testing SDD across a wider range of synthetic data generators.
  • Future Work: Whether the compute saving persists at production scale and in data mixtures where synthetic data is only part of the corpus remains open.
  • Future Work: The paper also proposes applying SDD beyond TSFMs to tabular foundation models.

A Distilled Losses

The appendix derives distilled objectives by taking conditional expectations of standard losses under the synthetic generator. These objectives preserve the target quantities while replacing realized future samples with distributional information.

  • Setup: The derivations fix a history and horizon, condition expectations on the observed history, and assume finite second moments.The conditional mean and standard deviation are used to standardize the future variable; experiments use Gaussian conditional laws.
  • Setup: Distilled losses aggregate independently across horizon indices, requiring only each future value’s marginal conditional law.Conditional expectation is linear, so the same aggregation applies whether the loss is summed or averaged over the horizon.
  • Squared Error: For squared error, distillation trains against the conditional mean up to a parameter-independent additive constant.The conditional variance contributes only the constant term s^2.
  • Pinball Loss: For pinball loss, the distilled objective is convex and is minimized at the conditional τ-quantile.Its derivative is F0(z) − τ, and its second derivative is f0(z)/s ≥ 0.
  • Cross-Entropy: For binned cross-entropy, distillation replaces the one-hot bin from one realized draw with the exact conditional bin distribution.The objective is the cross-entropy of the predicted bin probabilities against those conditional probabilities.

B Tractable Distilling Losses

The appendix expresses distilled losses for tractable generator families through a small set of conditional distribution functionals. This provides closed forms that can be tabulated and extended to new generator families.

  • Scope: The appendix derives distilled losses for four synthetic generator families and every loss listed in Table 1.
  • Tabulation: The tabulated grid factorizes, making it small enough to tabulate and usable by matching a generator-family row with a loss row.
  • Common representation: The generator enters the distilled loss through conditional mean, variance, CDF, and upper partial expectation functionals.The loss determines how these four functionals are combined.
  • Closed forms: Proposition 2 gives closed forms for distilled losses in terms of µ, s, F, and U under a finite-second-moment assumption.The pinball form is ℓdistill(ŷ) = U(ŷ) + (1 − τ)(ŷ − µ).
  • Quantile consequence: The pinball objective has derivative F(ŷ) − τ and therefore targets the exact conditional τ-quantile for any conditional law.Distillation does not change the quantity the quantile head is trained to learn.
  • Extensibility: A new generator family needs to provide a CDF and an upper partial expectation to join the closed-form table.For the lognormal case, these ingredients yield the required upper partial expectation before substitution into Proposition 2.

B.2 Which Published Synthetic Data Generators are Tractable

Published synthetic generators vary in whether their conditional forecast laws are available in closed form. Table 3 maps these generators to the tractable law families and corresponding distilled losses.

  • Fallbacks: For generators without closed-form laws, the paper uses either a Monte Carlo estimate of the distilled loss or the Status Quo loss.The final three rows require these fallbacks.
  • Distilled losses: Table 2 summarizes conditional forecast laws using four functionals and gives the corresponding distilled loss for each tractable family.The families are Gaussian except for the log-linear SDE, which is lognormal, and the general i.n.i.d. case.
  • Generator tractability: Table 3 identifies which synthetic generators used in recent TSFM pre-training provide the conditional forecast law required by SDD.The first three rows are closed-form, while the last three are not.
  • Generator provenance: Rows one to five of Table 3 are TempoPFN generators, which subsume KernelSynth and ForecastPFN, while the last row comes from Chronos.

C Experimental Details

The experiments use Gaussian-process synthetic series and Toto-2 models trained under controlled, paired configurations. SDD losses are computed from conditional forecast quantities, while held-out metrics provide the cross-arm comparison.

  • Data generation: Synthetic series are Gaussian-process draws of length T = 512, with ten kernel choices and hyperparameters sampled per series.The runs use one channel unless a sweep varies it.
  • Distilled objective: The Gaussian-process conditional future law is available in closed form, so SDD uses its marginal means and variances for the quantile objective.Only marginal conditional laws are needed for the per-time-point losses.
  • Training objectives: Status Quo uses realized squared-error or pinball targets, whereas SDD uses the corresponding conditional quantities; all reported results use the quantile head.The two arms share the same optimizer and training setup.
  • Comparison caveat: Cross-arm training objectives are not exactly comparable because full-series normalization weights correlate with the Status Quo residual but not SDD’s constant variance term.Consequently, cross-arm comparisons use held-out metrics rather than logged training objectives.
  • Model: Every result uses Toto-2 next-patch predictors with patch length P = 32, spanning five architectures from 4.1M to 2454.3M parameters.Models are rebuilt from published configurations and trained from random initialization.
  • Evaluation: Held-out next-token CRPS is evaluated on fresh generator seeds, with ten paired seeds sharing initialization and data order between objectives.Training uses 300,000 steps in the streaming study, with cumulative compute estimated as 6ND.

D Additional Results

Additional studies reuse the main experimental setup while varying the training context. Their reported gaps and speed-ups should be interpreted within their respective study designs.

  • Study design: The additional studies reuse the Appendix C setups for teacher forcing and factor sweeps.Appendix D.1 reports gaps, with negative values favoring SDD, whereas Appendix D.2 reports speed-ups.

D.1 Teacher Forcing, Streaming

Under teacher forcing, SDD improves convergence across all tested Toto-2 sizes, but its advantage is much smaller than under contiguous patch masking. The teacher-forcing comparison uses the same metric and paired experimental setup as the masking study.

  • Setup: Teacher forcing uses the same five architectures, quantile head, 300,000 steps, batch size B = 16, noise level σ = 0.25, and ten paired seeds as Section 3.The objective is scored at every next-patch position rather than on a contiguous masked span.
  • Interpretation: The smaller teacher-forcing advantage is attributed to gradients already averaging sampling noise across all T time points, leaving less variance for SDD to remove.Contiguous patch masking scores only the masked span and therefore averages fewer target draws per step.
  • Absolute performance: Teacher forcing also reaches lower absolute CRPS at every size because its unmasked next-token metric matches its training objective.Masked-span arms are evaluated outside the regime in which they train.
  • Figure 3: Figure 3 plots held-out next-token CRPS against cumulative training compute, using separate scales for each architecture and means over ten paired seeds.Colours encode training objective, while panel titles encode model size; panels are not comparable across architectures.

D.2 Factor Sweeps

The factor sweeps show that SDD’s compute advantage depends on noise, sequence length, and batch size, with larger gains at longer contexts and smaller batches. Under equal-data accounting, however, update-count differences largely explain the much larger apparent batch effect.

  • Experimental design: The sweep reports SDD speed-up as Status Quo compute divided by SDD compute to reach the same converged held-out next-token CRPS.Noise and sequence-length levels use B = 256 for 100,000 steps, while batch levels use 50,000 steps per level.
  • Observation noise: 1.32× at σ ≤0.25, easing to 1.22–1.23× from σ = 1 and then remaining flat through σ = 10.The decline across two decades of observation noise is real but small.
  • Sequence length: 1.26× at T = 128 rises monotonically to 1.34× at T = 1024.Longer contexts make the conditional distribution sharper and its moments more informative, increasing signal per step.
  • Batch size: 1.26× at B = 64 falls monotonically to 1.13× at B = 512.Batch averaging already suppresses realized-future sampling noise, leaving less variance for SDD to remove at larger batches.
  • Batch-size accounting: Under equal-data accounting, the apparent batch effect collapses from −3.5% at B = 64 to −0.01% at B = 1024.Most of this difference comes from update counts because B = 256 takes one quarter as many steps as B = 64.
Loading 2609.09586v1…