Estimating Time Series Foundation Model Transferability via In-Context Learning

• Fine-tuning every available TSFM on a target dataset is computationally expensive and time-consuming, motivating efficient model selection.Source: §1 · p. 1
• Existing statistical and meta-learning approaches are limited by fixed assumptions, training corpora, or predefined model sets, restricting generalization to evolving datasets and models.Source: §1 · pp. 1–2
• TIMETIC organizes historical model–dataset–performance observations in a tabular context and uses a tabular foundation model to predict target-model fine-tuned performance.Source: §1 · p. 2, Figure 1 · p. 2
• TIMETIC introduces entropy evolution across model layers as an architecture-agnostic model characterization that supports generalization across varied TSFMs.Source: §1 · p. 2
• Approximately 0.6 mean Spearman rank correlation and a 30% improvement over zero-shot ranking are reported across 10 datasets, 10 TSFMs, and 3 forecasting settings.Source: §1 · p. 2, Figure 1 · p. 2

• Most transferability metrics target classification and rely on categorical-output assumptions, while only a few extend to broader tasks through characteristic similarity, marginal likelihood, or regression error.Source: §2 · p. 3
• Meta-learning methods predict model performance from task features but generally select from predefined model pools, limiting their applicability to unseen models and datasets.Source: §2 · p. 3
• Representation-based and LLM-based selection methods face dependence on encoder quality or opacity, leaving scalable generalization across unseen TSFMs and datasets unresolved.Source: §2 · p. 3

3 METHODOLOGY

• TIMETIC defines transferability estimation as producing model scores without fine-tuning on the target dataset, so scores can identify the best-performing candidate.Source: §3 · p. 3
• The framework combines data characteristics and entropy-based model characteristics, constructs an offline context table from observed fine-tuning, and performs online prediction with TabPFN.Source: §3 · p. 3, Figure 2 · p. 4, §3.3 · p. 6

3.1 TIME SERIES CHARACTERIZATION

• Feature extraction generates over 700 statistical descriptors from sampled time windows, but redundancy motivates information-richness and non-redundancy-based selection.Source: §3.1 · p. 4
• Lower TotalVariance indicates greater feature discriminative power, and greedy selection yields a compact 20-feature representation whose information content is comparable to the full 30-feature set.Source: §3.1 · pp. 4–5, Figure 3 · p. 5

3.2 MODEL CHARACTERIZATION

• The entropy profile tracks entropy across model layers and offers an inference-only characterization that distinguishes diverse architectures without architectural constraints.Source: §3.2 · p. 5
• Entropy profiles differ systematically across TSFM families: encoder–decoder models show two peaks, while decoder-only models generally have higher entropy than encoder-only models.Source: Figure 3 · p. 5, §3.2 · p. 6

3.3 IN-CONTEXT TRANSFERABILITY ESTIMATION

• The offline context table concatenates 20 data and 6 model characteristics with zero-shot and fine-tuned performance, while online inference predicts target performance without gradient updates or retraining.Source: §3.3 · p. 6
• TabPFN supplies the in-context learner because its pretraining enables regression on unseen tabular tasks without fine-tuning, and the final transferability score averages predictions across target windows.Source: §3.3 · p. 6

4.1 TRANSFERABILITY ESTIMATION BENCHMARK

• The benchmark spans 10 diverse datasets, 10 TSFMs from five families, and three forecasting tasks evaluated under standard and few-shot regimes using weighted Kendall’s τw.Datasets cover four domains, five sampling frequencies, and varied temporal characteristics; models range from 10M to 500M parameters.Source: §4.1 · p. 7

4.2 PERFORMANCE EVALUATION

• TIMETIC consistently outperforms all baselines across short-, medium-, and long-horizon forecasting tasks, with higher weighted Kendall’s τw correlations.The benchmark averages results across 10 datasets under standard and few-shot regimes.Source: §4.2 · pp. 7–8, Table 1 · p. 7
• TIMETIC achieves the strongest linear alignment and highest Spearman correlation of 0.6 between estimated transferability scores and fine-tuned performance.This comparison is shown for the medium-horizon task.Source: Figure 4 · p. 8, §4.2 · p. 8
• TIMETIC maintains strong performance with only minor Kendall’s τw fluctuations when transferability is estimated from just 100 training windows.It continues to outperform all baselines in the few-shot setting.Source: §4.2 · p. 8

4.3 GENERALIZATION EVALUATION

• Across known-model/unseen-dataset, unseen-model/seen-dataset, and unseen-model/unseen-dataset scenarios, TIMETIC achieves consistently higher rank correlations than zero-shot performance.The method requires only a limited number of observed examples as context for unknown models on unseen datasets.Source: §4.3 · p. 8, Figure 5 · p. 8

4.4 ABLATION STUDY

• Adding statistical features improves performance until 20 features, after which redundant or noisy features cause a slight decline.The information captured by 20 features is nearly equivalent to that captured by 30 features.Source: §4.4 · p. 9
• Entropy profiles improve transferability estimation by about 0.1 for unseen datasets, approximately 0.2 for unseen models, and approximately 0.15 when both models and datasets are unseen.The profile captures similarities across model architectures and scales, supporting inference for unseen models.Source: §4.4 · p. 9
• Increasing the context table from 1,000 to 6,000 time windows substantially improves performance, while TIMETIC remains robust with only 100 windows.The result indicates scalability with more observed transfer processes and reliability under limited context.Source: §4.4 · p. 9

• TIMETIC consistently surpasses existing methods in model ranking, showing substantial improvements in correlation with fine-tuned performance.Source: §5 · p. 9

A.1 FEATURE SELECTION

• PCA reduces standardized features to two components, after which K = 100 clusters approximate equivalence classes for computing TotalVariance.TotalVariance is the average variance across non-empty clusters.Source: §A.1 · p. 14
• Greedy feature selection incrementally adds the feature that most reduces TotalVariance until marginal improvement becomes negligible.The procedure is designed to reduce epistemic uncertainty at each iteration.Source: §A.1 · p. 14

A.2 TABPFN

• TIMETIC uses TabPFN as its tabular in-context learner for regression over contextual transfer observations.TabPFN is pretrained on a large collection of regression tasks.Source: §A.2 · pp. 14–15
• TabPFN normalizes table cells by context-table statistics, embeds them linearly, and applies separate attention across columns and rows.It predicts a probability distribution over target values rather than a single point estimate.Source: §A.2 · p. 14
• TabPFN inference scales as O(n^2 + m^2) in computation and O(n + m) in memory for n samples and m features.For 10,000 rows and 10 columns, reported inference time is approximately 0.2 seconds.Source: §A.2 · p. 15

• The benchmark pipeline covers target dataset and model collection, unified fine-tuning, baseline selection, and evaluation protocol.Source: B BENCHMARK CONSTRUCTION · p. 15

B.1 TARGET DATASETS

• The benchmark includes 10 datasets from four domains, five sampling frequencies, and five characteristics: trend, seasonality, transition, stationarity, and shifting.The dataset diversity is intended to simulate real-world TSFM transfer scenarios.Source: §B.1 · p. 15
• Short-, medium-, and long-term tasks use forecast horizons of 64, 256, and 512 steps with context lengths of 256, 1024, and 2048.Each dataset uses 90% of observations for training and 10% for testing.Source: §B.1 · p. 16

B.2 MODEL ZOO

• The model zoo contains 10 TSFMs from five families, spanning encoder-only, decoder-only, encoder-decoder, dense, sparse, and varied tokenization designs.Model sizes range from 8 million to 500 million parameters.Source: §B.2 · pp. 16–17

B.3 GROUND TURTH

• Ground-truth transfer performance is obtained by fine-tuning every model under one unified strategy before evaluating rankings.All model parameters are fine-tuned for one epoch with AdamW at a constant learning rate of 1e-5.Source: §B.3 · p. 18

B.4 BASELINES

• Baselines include embedding-based statistical scores, a meta-learner using data and model features, and zero-shot MASE performance.The meta-learner predicts fine-tuned performance from data characteristics, entropy profiles, and zero-shot performance.Source: §B.4 · pp. 18–19

B.5 EVALUATION METRICS

• Weighted Kendall’s tau measures ordinal association between ranked lists while assigning different importance to item pairs.Its value ranges from −1 for inverse ranking to 1 for perfect agreement, with 0 indicating no ordinal correlation.Source: §B.5 · p. 19
• Spearman’s rank correlation measures the monotonic association between two ranked lists using differences between item ranks.Its value ranges from −1 to 1, with 0 indicating no monotonic correlation.Source: §B.5 · p. 19
• MASE evaluates forecast accuracy relative to a naive baseline using a scale-free error measure.A MASE value below 1 indicates that the model outperforms the naive baseline.Source: §B.5 · p. 19
• The evaluation compares transferability estimation methods across short-, medium-, and long-term forecasting under standard and few-shot regimes.Tables D and E cover performance comparisons, while Tables F and G report Spearman rank correlations.Source: Table D · p. 20, §B.5 · pp. 20–21

C.1 PERFORMANCE EVALUATION USING WEIGHTED KENDALL TAU

• TIMETIC achieves the highest correlations on most datasets and consistently outperforms all baselines across standard and few-shot evaluations.Its performance is weaker on sz taxi but consistently strong on bitbrains rnd.Source: §C.1 · p. 22
• Forecast horizon affects transferability estimation, with performance fluctuations across short-, medium-, and long-term forecasting within the same dataset.The results identify forecast horizon as an important factor influencing TSFM performance and ranking.Source: §C.1 · p. 22

C.2 PERFORMANCE EVALUATION USING SPEARMAN CORRELATION

• About 30% improvement over zero-shot performance is achieved on average by TIMETIC using richer time series features and model characterization.The comparison covers short-, medium-, and long-term forecasting under both standard and few-shot regimes.Source: §C.2 · p. 22
• Zero-shot performance provides a relatively strong baseline, with higher correlation than the other evaluated metrics.Spearman correlation measures global monotonic consistency between rankings.Source: §C.2 · p. 22

D UNCERTAINTY ANALYSIS

• The optimal pointwise regressor is lower-bounded by Var(y|x), the aleatoric uncertainty in actual fine-tuned performance.The performance-estimation task models p(y|x), where y is fine-tuned performance and x is the raw time series.Source: D UNCERTAINTY ANALYSIS · p. 22
• Using feature representations ϕ(x) raises the error lower bound to Var(y|ϕ(x)) because distinct series can share the same representation.This additional epistemic uncertainty prevents the regressor from distinguishing states with identical features but different raw series.Source: D UNCERTAINTY ANALYSIS · p. 22
• TotalVariance serves as a practical proxy for epistemic uncertainty and guides context-table construction by favoring informative features.The analysis also extends the uncertainty argument to rank correlations when multiple outcomes share identical feature representations.Source: D UNCERTAINTY ANALYSIS · p. 22

E USE OF LARGE LANGUAGE MODELS

• Large language models were used only to improve writing clarity and readability, while the substantive research was conducted by the authors.The stated substantive contributions include conceptualization, model design, experimentation, and analysis.Source: E USE OF LARGE LANGUAGE MODELS · p. 22