Source-linked AI summary
TW3Cast: A Frozen Router of Lightly Fine-Tuned Foundation Models for Time-Series Forecasting on GIFT-Eval, Selected Entirely on the Training Split
Nathan Thierry, Andre-Louis Rochet
TL;DR
Choosing a forecaster per time-series configuration is challenging when practitioners face multiple public foundation models, sizes, context policies, and fine-tuning options. TW3Cast freezes a training-split routing table over lightly fine-tuned experts, using train-side selection and guarded mechanisms. It reaches position 3 of 130 by mean MASE rank on GIFT-Eval as of 2026-09-14.
Problem
Practitioners must choose among public foundation models, sizes, context policies, and fine-tuning options, while no model served alone reaches the top of GIFT-Eval.
Source: §1 · p. 1Method
Results
Position 3 of 130 by mean MASE rank is reached on GIFT-Eval, with a mean rank of 19.4.
Source: §9 · p. 7Takeaways & Limitations
Admission by beating the tournament makes improvement monotone on the selection criterion and prices failed experiments at candidate-training cost.
Source: §3 · p. 3The dual MASE and CRPS criterion does not select for calibration, and weighted quantile loss rank (position 8) lags MASE rank (position 3).
Source: §11 · p. 10
Abstract
from arXiv · showhide
TW3Cast is a time-series forecasting system that reaches position 3 of 130 entries on the GIFT-Eval benchmark by mean MASE rank, as of 2026-09-14. The two entries above it belong to the leaderboard's agentic category, multi-step systems that use agents or language models to reason about, generate or select forecasts. TW3Cast runs no agent and no language model. Its selection is a table computed once on the training split and then frozen, and its experts are public foundation models lightly fine-tuned on those training splits. For each of the 97 dataset, frequency and horizon configurations, the table serves one of four modes: a specialist, which is a LoRA or full fine-tune of Chronos-2, TiRex or Toto whose training data was cleaned and enriched by explicit rules; a quantile blend that contains a specialist; a blend of base models; or a selection tournament played on a backtest carved from the training split. Every decision in the table was taken on that backtest. A specialist is admitted the moment it beats the tournament there, so a candidate costs a few megabytes and minutes of GPU time, and a failed candidate changes nothing. Three guarded mechanisms protect the selection from its own biases: a dual accuracy and calibration criterion, an asymmetric margin against candidates that saw the series during training, and conservative per-window gates. The selection rules themselves were chosen inside a temporal meta-backtest. The best base model served alone reaches a mean MASE rank of 33.8, the tournament served on every configuration reaches 38.0, and the full router reaches 19.4. The routing table, the expert index, the pinned base-model revisions, the submitted score file and the dated snapshot of the public scores are released, and every leaderboard number in this paper regenerates from them by one script.
1 Introduction
TW3Cast addresses the choice of forecaster for each data source by using its recent history to route among public foundation models. Its frozen, measured routing differs from learned selectors and inference-time agentic routing, trading expressiveness for auditability and no inference-time overhead.
1 Introduction
- Per-source recent history informs which model should forecast, amid choices among public foundation models, model sizes, context policies, and fine-tuning.GIFT-Eval aggregates 97 configurations over 24 datasets, and no model served alone reaches the leaderboard top.Source: §1 · p. 1
- The frozen table assigns mixture weights per configuration, with explicit decisions measured on a training-split backtest and no further selection at prediction time.This makes the router auditable and improvable row by row.Source: §1 · p. 2
- Position 3 of 130 by mean MASE rank is reached through train-side selection, lightly fine-tuned experts, and guarded selection mechanisms.The system is the best entry without an agent or language model; its admission rule accepts candidates that beat the tournament on the backtest.Source: §1 · p. 2
2 Related work
- Unlike agentic leaderboard systems, TW3Cast uses no language model, freezes routing from the training split, and obtains experts through light fine-tuning.It shares with those systems routing among public foundation models per data source.Source: §2 · p. 2
- Equal weights are used within blends, reserving estimation for choosing blend members rather than estimating weights.This responds to the forecast-combination puzzle in which estimated weights can lose to equal weights.Source: §2 · p. 3
- Unlike FFORMS and FFORMA selectors that learn across series, TW3Cast selects per configuration from its own history and freezes the measured routing table.This trades expressiveness for auditability and zero inference-time overhead; it also uses guard rails against selection bias.Source: §2 · p. 3
3 Problem statement
The task is to choose a forecaster for each benchmark configuration using only its training split, minimizing mean per-configuration rank. Candidate specialists are admitted only when they beat the tournament on a backtest under a dual scoring criterion.
3 Problem statement
- For each of 97 configurations, choose a single forecaster, quantile blend, or per-window policy using only training data to minimize mean per-configuration rank.A configuration is a dataset, sampling frequency, and horizon-class triple; windows are scored by MASE for point accuracy and mean weighted quantile loss for calibration.Source: §3 · p. 3
- A specialist is designated once it beats the tournament champion on the backtest under the dual criterion, with the specialist preferred under comparable scores.Otherwise, the configuration retains the tournament serving mode.Source: §3 · p. 3
- Admission makes improvement monotone on the selection criterion, while a failed experiment costs only the candidate's training.This claim is limited to the selection criterion.Source: §3 · p. 3
4 The router
The router is a released table that maps each configuration to experts and mixture weights, encoding specialist, blend, or tournament serving. Its configurations are distributed across these modes, with specialists drawn from three architecture families.
4 The router
- The routing table has one row per configuration and expert, assigns mixture weights, and pairs with an expert index listing each expert's architecture family.Weights in each row sum to one; a designated specialist has weight one.Source: §4 · p. 3
- A configuration name resolves by table lookup to a specialist, quantile blend, or tournament mode; per-window gates bound tail risk before nine deciles are output.Blends may mix specialists and base models, and inference uses a sorted weighted average of quantiles.Source: Figure 1 · p. 4, §4 · p. 4
- The released router serves 16 configurations with a specialist, 32 with a blend containing a specialist, 22 with a base-model-only blend, and 27 with the tournament.Its 43 specialists comprise 19 Chronos-2, 13 TiRex, and 11 Toto experts.Source: §4 · p. 4, Figure 2 · p. 4
5 The decision protocol
The decision protocol scores candidates on recent windows carved from the training split, while the remaining training data trains specialists. Selection rules are themselves validated by a temporal meta-backtest before the router is frozen ahead of test evaluation.
5 The decision protocol
- Recent training-split windows supply every selection decision, and the resulting router is frozen before any test window is read.The remaining training data is used to train experts.Source: Figure 3 · p. 5
- Roughly 100 to 600 backtest windows per configuration are scored using benchmark metrics, and the remaining training data trains specialists.Windows are sliced from the end of the training split; MASE uses the real context before padding, while CRPS is approximated by mean pinball loss over nine quantiles.Source: §5 · pp. 4–5
- A temporal meta-backtest selects on older windows and scores the choice on recent ones across hundreds of draws, validating the selection rule itself.Margin, tie-break, and gate-threshold iterations were carried out within this protocol without evaluation data.Source: §5 · p. 5
6 Building the experts
The section argues that fine-tuning value depends on explicit, inexpensive data preparation, and admits each resulting specialist only if it beats the tournament on the backtest.
Cleaning
- Data-rich configurations favor cleaned solo specialists, while data-poor configurations benefit from enriched sibling datasets.The section reports both outcomes in a 21-series weather example, depending on whether the configuration is data-rich or data-poor.Source: §6 · p. 5
Dosed fusion
- For data-poor configurations, dosed fusion typically mixes three parts target data with one part sibling data, keeping the target over-weighted.Families are formed first by frequency and horizon, then by domain; poor families may borrow from rich siblings.Source: §6 · p. 5
Controlled windows
- Training windows use variable-length contexts and robust residual normalization, and optimize pinball loss on the benchmark’s nine deciles.The target is the full horizon or next patch depending on architecture; learning rates are 10−4 for LoRA and 10−5 for full fine-tunes.Source: §6 · pp. 5–6
Three architectures and cross blends
- Candidates enter the router only by beating the tournament on the backtest, using minutes of training rather than global retraining.The admission rule applies to every candidate produced by the preparation pipeline.Source: §6 · pp. 5–6
- Chronos-2, TiRex, and Toto are specialized with native recipes, and 42 of the 54 released blends mix architectures.Blends are sorted quantile means of their members.Source: §6 · p. 6
7 The tournament and its guard rails
The tournament selects a default using backtest performance, with criteria and gates designed to guard against misleading wins and per-window failures.
Stage 1, a selection defended against its own biases
- The selection requires both MASE and CRPS, since degenerate quantiles can win on MASE while being probabilistically useless.Near-ties below a 3% gap are settled on CRPS.Source: §7 · p. 6
- Locally trained candidates must beat the best generic score by a 12% margin and cannot be worse on CRPS.The margin addresses backtest scores flattered by candidates that saw the series during training.Source: §7 · p. 6
- When the blend of the two largest pool architectures is within 5% of the winner, the blend is served instead.The section treats a narrow win as potentially backtest noise and blending as safer under uncertainty.Source: §7 · p. 6
Stage 2, a conservative per-window gate
- A per-window switch requires the default to exceed its error threshold and a challenger to win by 10% on average, on both segments, and on CRPS.The gate checks the final horizon-length segment and a middle segment, and is designed to switch rarely.Source: §7 · p. 6
- A regime-transition guard blocks switching when recent context dormancy differs from the segments, as can happen at dawn and dusk in hourly series.Source: §7 · p. 6
Safety nets for designated experts
- Designated specialists have a separate high-threshold catastrophe gate, while imported predictions are rejected if median alignment correlation falls below 0.30.The alignment guard fired three times on real misalignments and never on an aligned import.Source: §7 · p. 7
8 Evaluation protocol
The paper evaluates forecasts with the official GIFT-Eval harness and ranks entries by average per-configuration ranks. In the dated leaderboard snapshot, TW3Cast ranks third by mean MASE rank, with results also comparing its components, tournament, and specialists.
8 Evaluation protocol
- Scores use the official GIFT-Eval harness and leaderboard parameters, and the dated snapshot ranks complete public entries by average ranks over 97 configurations.The authors validated their harness installation by reproducing a published reference model to three decimals and checked submitted artifacts against MASE and weighted quantile loss.Source: §8 · p. 7
9 Results
- Position 3 of 130 by mean MASE rank, with mean rank 19.4, places TW3Cast behind only entries declared agentic by their authors.By mean weighted quantile loss rank, TW3Cast is position 8 with mean rank 24.6, so calibration lags point accuracy.Source: §9 · p. 7, Table 1 · p. 8, Figure 5 · p. 9
- The best pool member alone, Toto 2.0 2.5B FT, sits at position 23 with a mean MASE rank of 33.8.Against a per-configuration test-MASE oracle, TW3Cast is better on 37 configurations, within five percent on 60, and more than ten percent worse on 30.Source: §9 · p. 7, Table 2 · p. 8
- Served on every configuration, the tournament reaches a mean MASE rank of 38.0, behind the best single member at 33.8.On the 27 configurations actually assigned to the tournament, its rank is 19.8 versus 28.1 for the best member.Source: §9 · p. 8
- On the other 70 configurations, a specialist or blend beats tournament-only serving on 69, with a median MASE gain of 2.0 percent.The router combines the tournament where it wins with specialists or blends where they perform better.Source: §9 · p. 8
- Across several configurations, all trained fine-tuned candidates lost to the tournament and were discarded; a cleaned-solo Chronos-2 LoRA won on a 10-minute weather configuration.The 48 configurations served by or with a specialist have designated experts that beat the tournament on the backtest by construction.Source: §9 · p. 8
10 Deployability and accretion
TW3Cast deploys as a frozen lookup table with expert forward passes and sorted quantile averaging, and can grow through locally admitted candidates without global retraining.
10 Deployability and accretion
- Inference uses a table lookup, expert forward passes, and a sorted weighted average of quantiles, with no agent or language model.Source: §10 · p. 9
- A candidate LoRA adapter takes 500 to 2,000 steps and two to seven minutes on a single GPU, and is admitted only if it beats the tournament on the backtest.A failed candidate changes nothing, and integrating, replacing, or retiring an expert changes one table row.Source: §10 · p. 9
- On a new data source, the tournament can build a backtest from available history and serve a default with the same guard rails, without training.Specialization can be tried on that source’s history, with admission decided by the same rule.Source: §10 · p. 9
11 Threats to validity
Validity is bounded by a dated leaderboard snapshot and train-end temporal assumptions; selection safeguards and artifact serialization introduce further stated limits.
11 Threats to validity
- A dated leaderboard snapshot fixes the comparison set, not future standings, so a later reader should expect the reported position to change.Source: §11 · p. 9
- Figure 6 compares per-configuration MASE with the best pool member across 97 configurations; negative bars mark cases where TW3Cast beats that per-configuration oracle.Source: Figure 6 · p. 10
- An abrupt regime change at the train–test boundary would defeat both the end-of-training backtest and the temporal meta-backtest.The recency bias is an assumption about regime continuity, not a theorem.Source: §11 · p. 10
- Selection margins and thresholds can overfit validation data; freezing constants chosen inside the temporal meta-backtest does not remove their dependence on this benchmark’s frequency and domain mix.None of the constants was tuned on official evaluation results.Source: §11 · p. 10
- Position 8 for weighted quantile loss rank trails position 3 for MASE rank; the dual criterion filters poorly calibrated winners but does not select for calibration.A CRPS-first admission rule is identified as an unevaluated variant.Source: §11 · p. 10
- One base-library serialization path drifted by under one percent; published scores therefore use reloaded artifacts, and one expert is shipped as verified forecast quantiles rather than retained seed-averaged weights.The quantile file reproduces forecasts but not training.Source: §11 · p. 10
12 Reproducibility
The release supports both exact regeneration from files and fresh inference with pinned models and a GPU. TW3Cast reaches position 3 of 130 on GIFT-Eval, with small verified changes presented as its practical consequence.
12 Reproducibility
- The release includes the routing table, expert index, pinned base-model revisions, reference inference script, expert checkpoints, submitted result file, and dated public-score snapshot.Source: §12 · p. 11
- Exact regeneration reproduces every paper table, figure, and number from the release without a model call; fresh regeneration runs inference and official scoring with pinned models, checkpoints, and one GPU.Source: §12 · p. 11
13 Conclusion
- Position 3 of 130 by mean MASE rank on GIFT-Eval was reached on 2026-09-14 without an agent or language model at inference.Source: §13 · p. 11
- Admission by beating the tournament makes each expert upgrade individually verified at minutes per candidate, supporting a stream of small changes instead of periodic retraining.The paper identifies this accretion property as the expected practical consequence that matters most.Source: §13 · p. 11
A Constants and thresholds
The appendix records frozen constants and the released routing table, including configuration modes, blend members, and abbreviation conventions.
A Constants and thresholds
- Every released constant was chosen inside the train-side protocols and then frozen.Source: Table 4 · p. 14
B The released routing table
- The routing-table abbreviations identify Chronos-2, TimesFM 2.5, Toto variants, TurkForecast’s Chronos-2 LoRA, and two TiRex releases.C2 denotes Chronos-2; TFM denotes TimesFM 2.5; Toto, Toto-uni, Turk, TiRex, and TiRex-2 denote the specified variants.Source: B The released routing table · p. 14
- Tables 5 and 6 present the released routing table’s two halves, listing configurations and their served modes and blend members; blend members carry equal weights.Source: Table 5 · p. 15, Table 6 · p. 16