PaperlayerView exact revision on arXiv

Source-linked AI summary

Latent Inference-Time Guidance of Time Series Foundation Models

Chloé Hashimoto-Cullen, Amaury Durand, Laurent Bozzi, Benjamin Guedj, Yannig Goude, Sylvain Le Corff

arXiv:2609.38058v1stat.MLcs.LGstat.ME

TL;DR

TSFM forecasts vary with prompt choices and have complementary strengths, while a general probabilistic framework for latent inference-time guidance in forecasting is absent. The paper introduces a time-dependent latent guidance mechanism for combining frozen TSFM forecasts, and reports empirical benefits across datasets spanning multiple domains and sampling frequencies.

  • Problem

    TSFM forecasts are sensitive to prompt choices and have complementary strengths, but a general probabilistic framework for latent guidance in time-series forecasting is lacking.

    Source: §1 · pp. 1–2

  • Method

    LITiG-TSFM combines expert forecasts through a structured, time-dependent latent state while keeping the underlying foundation models frozen.

    Source: §1 · p. 2

  • Results

    Across datasets spanning multiple domains and sampling frequencies, LITiG-TSFM demonstrates empirical benefits and is competitive with its baseline methods.

    Source: §1 · p. 2, §4.2 · p. 8

  • Takeaways & Limitations

    The lightweight guidance mechanism can be learned without access to or fine-tuning of TSFM parameters.

    Source: §1 · p. 2

    Some assumptions used to obtain the results may be too restrictive for real-life datasets.

    Source: §5 · p. 10

Abstract

from arXiv · show

Time Series Foundation Models (TSFMs) currently provide state-of-the-art results in forecasting tasks. They are available out-of-the-box and rely on in-context learning to make their predictions, which makes the quality of their performance highly sensitive to the user-selected lookback, covariates, horizon and training data distributions. In practise, the quality of the forecasts are variable but complementary, which highlights the need for a principled ensembling approach, rather than selecting the best context. This paper introduces Latent Inference-Time Guidance for TSFMs, which adaptively combines a pool of TSFM forecasts through a time-dependent latent space with independent components. The framework comes equipped with identifiability and reconstruction guarantees, whilst maintaining the off-the-shelf aspect of foundation models. We provide experiments on datasets at various frequencies and from multiple domains: these show that the approach is competitive with traditional ensembling approaches.

1 INTRODUCTION

TSFM forecasts vary with prompt choices and often have complementary strengths, motivating adaptive combination rather than selecting one model. The paper introduces a probabilistic latent-guidance framework that combines frozen TSFM forecasts and reports empirical benefits across varied datasets.

1 INTRODUCTION

  • Different TSFMs and configurations can have complementary strengths, making principled, adaptive forecast combination the central challenge.Forecast accuracy is sensitive to covariates, lookback, and horizon choices.Source: §1 · p. 1
  • The paper proposes inference-time guidance because no general probabilistic framework for latent guidance in time-series forecasting was known to the authors.The approach learns a lightweight mechanism to steer or combine expert outputs using downstream observations rather than modifying the foundation models.Source: §1 · p. 2
  • A time-dependent latent state weights or corrects each expert over time, with an independent-component structure supporting identifiability and reconstruction guarantees.The underlying foundation models remain frozen; only the lightweight guidance mechanism is learned.Source: §1 · p. 2
  • The contributions include a probabilistic guidance framework, identifiability and reconstruction-stability guarantees, and empirical evaluation across domains and sampling frequencies.The framework adaptively combines and corrects forecasts from frozen TSFMs.Source: §1 · p. 2

2 BACKGROUND AND RELATED WORKS

The background traces the development of foundation models for tabular and time-series forecasting, then outlines in-context prediction, variational latent modeling, and structured nonlinear ICA. These foundations frame the paper’s use of TSFM forecasts and structured latent states.

Foundation models

  • Transformer-based architectures and large-scale pretraining supported competitive tabular foundation models, while encoder and decoder designs use reconstruction and causal modeling, respectively.The account notes that early tabular foundation models were univariate, limiting forecasting capacity.Source: §2 · p. 2
  • TSFMs use in-context prediction on real-world data after training on synthetic data, and forecast performance can vary with training data.The passage also notes that TSFMs may incorporate covariates and support multivariate forecasting.Source: §2 · p. 3

Variational AutoEncoders

  • Variational autoencoders approximate conditional distributions in latent-data models using a prior over latent variables and a conditional decoder.The variational approach uses an encoder to approximate the conditional distribution of latent variables given observations and covariates.Source: §2 · p. 3

Structured Nonlinear Independent Component Analysis (ICA)

  • Structured nonlinear ICA assumes independent latent components and adds conditions including stationarity, conditional independence, an injective mixing function, and i.i.d. noise.The cited account describes these assumptions as part of the structured nonlinear ICA framework.Source: §2 · p. 3

3 LATENT GUIDANCE OF FOUNDATION MODELS

LITiG-TSFM uses structured, time-dependent latent variables as a probabilistic interface for guiding pretrained foundation models at inference time, without retraining them. The section establishes identifiability and reconstruction guarantees under stated conditions and describes variational training and forecasting.

Latent guidance

  • Each latent component has its own dependency structure while remaining conditionally independent, and the model parameterizes transitions and emissions with neural maps.The latent prior factorizes across components.Source: §3 · p. 4
  • LITiG-TSFM uses a nonlinear ICA perspective and time-dependent latent processes as an inference-time interface for guiding pretrained models without retraining.The latent processes capture sequential dependencies and allow adaptive conditioning on past observations or covariates.Source: §3 · p. 4

Variational training

  • The model is trained by maximizing an ELBO decomposed into observation reconstruction, latent dynamical consistency, and posterior-to-prior regularization terms.A standard variational family uses Gaussian distributions parameterized by neural networks.Source: §3 · pp. 4–5

Inference and forecasting

  • Forecasting propagates learned latent dynamics under the generative model, while a VAMP-type inference-time posterior adaptation can refine forecasts without retraining pθ.The practical approximation initializes the latent state from its variational posterior given observations and covariates to date.Source: §3 · p. 5

Identifiability results

  • Under A1–A4, the denoised latent guidance signal is identifiable from noisy observations up to translation; identifying ICA components additionally requires decoder conditions.The section distinguishes identifiability of the denoised signal from that of the ICA components.Source: §3 · p. 5

Reconstruction guarantee

  • Assumption B2 does not hold on an unbounded state space, though truncating latent spaces can usually satisfy it.B2 is used for quantitative bounds on approximation and control of joint smoothing distributions.Source: §3 · p. 6
  • With B1–B3 and uniformly close backward variational kernels, smoothing and classical reconstruction errors grow at most linearly with the time horizon.The excess reconstruction risk relative to the oracle smoother is controlled linearly by backward-kernel approximation error, so local errors do not compound exponentially.Source: §3 · p. 6

4 EXPERIMENTS

Experiments span multiple domains, sampling frequencies, and noise levels, comparing LITiG-TSFM with TSFM and aggregation baselines. Results report competitive performance and robustness when weaker experts are added.

4 EXPERIMENTS

  • The evaluation covers varied domains, frequencies, noise levels, and seasonality, uses RMSE and MAE, and reports means and standard deviations for nondeterministic methods run ten times.Datasets range from hourly to monthly timesteps.Source: §4 · pp. 6–7, Table 1 · p. 7

4.1.1 IDEAL SETTINGS BASELINES

  • The oracle mixture is an inaccessible performance floor because it requires ground truth to select the best expert at each timestep.The high-performing expert set uses TabICLv2 forecasts selected on the validation fold and evaluated on the test fold.Source: §4.1.1 · p. 7

4.1.2 MIXTURE OF EXPERTS BASELINES

  • Baselines include ML-Poly, which learns linear forecast weights and has drift-robustness guarantees, and a Kalman filter with A fixed to identity and Q and R learned by expectation maximisation.The Kalman filter's process and observation noise parameters are learned using expectation maximisation.Source: §4.1.2 · pp. 7–8

4.2 RESULTS

  • Washington Bicycle Share and Mauna Loa expose weak TSFM forecasts, associated respectively with noisy high-frequency data and a trend continuing beyond the training domain.The paper presents these datasets as use cases for mixture-of-experts methods.Source: §4.2 · p. 8, Figure 2 · p. 9
  • LITiG-TSFM is competitive with baselines on datasets with strong TSFM forecasts and remains robust as poor experts are added.In the experts ablation, TSFM error grows with added weaker experts, while LITiG-TSFM does not deteriorate.Source: §4.2 · pp. 8–9, Figure 1 · p. 8, Figure 3 · p. 9
  • For multiple-series or high-frequency datasets, Kalman filtering can take longer than training and prompting LITiG-TSFM, which also requires no warm-up window.Without a warm-up window, forecasts become useful in fewer inference steps and initial steps need not be discarded.Source: §4.2 · p. 9

5 DISCUSSION, LIMITATIONS AND CONCLUSION

LITiG-TSFM combines TSFM forecasts through latent structured state-space modelling and provides adapted reconstruction guarantees and novel forecasting guarantees. The authors note limitations in the assumptions and the choice of TSFM used in experiments.

5 DISCUSSION, LIMITATIONS AND CONCLUSION

  • LITiG-TSFM combines TSFM forecasts through latent structured state-space modelling, with guarantees on latent dynamics, reconstruction, and forecasting.Its reconstruction guarantee uses a backward factorisation, whereas the implementation uses forward factorisations.Source: §5 · p. 9
  • Some assumptions used to obtain the results can be too restrictive for real-life datasets.The experiments use TabICLv2 to create experts, although the authors say the same experiments could use any existing out-of-the-box TSFM.Source: §5 · p. 10
  • The experiments use TabICLv2 as an example for creating experts, but the authors say the same experiments could use any existing out-of-the-box TSFM.Source: §5 · p. 10
  • The paper reports using generative AI for method implementation, dataset cleaning and reformatting, qualitative and thematic analysis, and software code.The authors state that AI tools were not used to develop theoretical models, formulate mathematical claims, write proofs, design experiments, or interpret results.Source: AI USE STATEMENT · p. 10
  • The reproducibility statement points to assumptions and results in Section 3, proofs in Appendix A, code in Section 4, and implementation details in Appendix B.Source: REPRODUCIBILITY STATEMENT · p. 10

A.1 DERIVATION OF THE EVIDENCE LOWER BOUND

The derivation sets up a joint model for targets, covariates, and expert forecasts, then uses variational inference to approximate an intractable posterior and optimize an ELBO.

A.1 DERIVATION OF THE EVIDENCE LOWER BOUND

  • The derivation defines target data, covariates, and N expert forecasts, then uses x1:T as shorthand for covariates together with the forecasts.Source: §A.1 · p. 15
  • The method maximizes the ELBO over model and variational parameters to learn the joint distribution.Source: §A.1 · p. 15
  • The latent prior is factorized across the initial state and transitions, while the Gaussian decoder factorizes observations across time.The prior uses a Gaussian initial state and transition; the decoder models each y_t using z_t and the expert forecasts F_t.Source: §A.1 · p. 15
  • Because the posterior is intractable, the method approximates it with a variational family that factorizes sequentially over latent states.The initial factor conditions on x1 and y1; later factors condition on past covariates and observations and the previous latent state.Source: §A.1 · p. 15

A.2 DERIVATION OF THE FORECASTING DISTRIBUTION

The section describes how the model approximates the predictive expectation for a new observation using its trained variational family, with a VAMP-like alternative for latent-state expectations.

A.2 DERIVATION OF THE FORECASTING DISTRIBUTION

  • The target is a conditional expectation for a new observation given past observations and covariates, approximated using the trained variational family.Source: §A.2 · p. 16
  • The variational distribution over the latent trajectory is expressed through a terminal smoothing distribution and backward conditional distributions.Source: §A.2 · p. 16
  • For the latent-state expectation, the method can use the variational approximation or a VAMP-like sampler.The expectation is conditioned on observations and covariates through time t.Source: §A.2 · p. 16

A.3 PROOF OF PROPOSITION 3.2

The proof treats the proposed model as a general state-space model conditional on covariates and forecasts, and derives bounds using stated assumptions and backward-kernel constructions.

A.3 PROOF OF PROPOSITION 3.2

  • Conditional on covariates and forecasts, the proposed nonlinear ICA model is treated as a general state-space model, allowing use of a cited proposition under strong mixing assumption B2.The proposition is applied using an additive state functional.Source: §A.3 · p. 16
  • The proof selects a backward-kernel construction by setting each tilde q_k to phi_theta,k and then suppresses covariate notation for clarity.Source: §A.3 · p. 17
  • The proof uses assumption B3 after the backward-kernel term is zero.Source: §A.3 · p. 17
  • Assumption B1 bounds the squared residual between y_t and h_theta(F_t, z_t), which is used to derive a constant bound.Source: §A.3 · p. 17

B EMPIRICAL DETAILS

The empirical details examine LITiG-TSFM across datasets and prompting setups, with qualitative samples, compute comparisons, and an architecture ablation. The samples describe corrections to expert forecasts, while the ablation favors an attention encoder and MLP decoder.

TSFM prompts for expert creation

  • TabICLv2 generates experts through single-step forecasts from the training-fold context, with ground truth replacing forecasts at the selected prompting frequency; mean and raw quantile outputs are used.Hyperparameters are selected separately for each dataset.Source: B EMPIRICAL DETAILS · pp. 17, 19

Datasets

  • London Smart Meter experiments aggregate household electricity consumption by Acorn segments K–O to test noisy data and forecasting across multiple categories.The dataset records household consumption in London from 2012 to 2014.Source: B EMPIRICAL DETAILS · p. 18
  • NN5 provides weekly demand for 111 UK ATMs, with date-derived month and week covariates, testing multiple-series learning with little covariate information and substantial noise..Source: B EMPIRICAL DETAILS · p. 18
  • Mauna Loa monthly CO2 is differenced to avoid testing on out-of-domain data, and its test-fold distribution differs from the training-fold distribution.The series has strong seasonal and trend components.Source: B EMPIRICAL DETAILS · p. 18
  • Washington Bicycle Share provides hourly rentals with weather and calendar covariates; the experiments use lags of subscribed, casual, and total users, and normalize total users between 0 and 1.The dataset has high frequency and noise.Source: B EMPIRICAL DETAILS · pp. 18–19

Inference-time compute

  • Kalman must be refit for each London Smart Meter and NN5 series, whereas LITiG-TSFM is trained once and then runs inference across series.Table 7 compares TSFM prompting, Kalman expectation maximisation, and LITiG-TSFM training and inference costs.Source: B EMPIRICAL DETAILS · p. 20, Table 7 · p. 20

Qualitative samples

  • On London Smart Meter, LITiG-TSFM corrects experts’ low forecasts and captures peaks and valleys more often than Kalman; Washington and NN5 samples also show peak-and-valley fitting and improved handling of expert forecasts.For NN5, ML-Poly and Kalman either oversmooth the mixture or fail to learn the pattern consistently; LITiG-TSFM bridges these approaches.Source: Figure 4 · p. 20, B EMPIRICAL DETAILS · pp. 20–21, Figure 5 · p. 21

Architecture ablation

  • The reconstruction ablation reports stronger performance with an MLP decoder and an attention encoder; the MLP decoder also removes the constraint tying latent dimension to expert count.The dot-product decoder instead constrains latent dimension to the number of experts.Source: B EMPIRICAL DETAILS · pp. 21–22

Related papers

Loading 2609.38058v1…