Source-linked AI summary
LiveHouse-TS: An Open-world Living Benchmark for Time Series Foundation Models
Haomin Wen, Ziyu Zhou, Qingxiang Liu, Siru Zhong, Yuxuan Liang
TL;DR
Static TSFM benchmarks freeze historical test windows, limiting evaluation of behavior under evolving conditions. LiveHouse-TS evaluates models prequentially on streaming future data and finds that static rankings differ under live evaluation.
Problem
Static benchmarks use frozen test windows, limiting evidence about model behavior under non-stationarity, concept drift, and sudden exogenous shifts.
Method
LiveHouse-TS is an open-world benchmark that evaluates TSFMs prequentially on continuously streaming observations with metrics updated as new data arrive.
Results
Static benchmark rankings differ from LiveHouse-TS rankings, although current TSFMs generalize well for zero-shot forecasting on real future data.
Takeaways & Limitations
LiveHouse-TS indicates that strong static-benchmark performance may not translate directly to practical deployment under changing conditions.
Takeaways & Limitations
Geographic, domain, availability, and measurement biases in selected streams may affect scores and rankings, which should not be interpreted as universal superiority or downstream fairness.
Abstract
from arXiv · showhide
Time Series Foundation Models (TSFMs) have recently emerged as a highly promising paradigm for cross-domain zero-shot forecasting. However, existing evaluation protocols predominantly rely on static benchmarks with fixed historical test windows. While these benchmarks provide a valuable baseline snapshot, they evaluate an average performance on a fixed history, failing to capture how models behave in continuously evolving real-world environments characterized by seasonal variations, distribution shifts, and unexpected events. To bridge this gap, we introduce LiveHouse-TS, the first open-world living benchmark infrastructure for TSFMs. By evaluating models prequentially on real future data in open-world environments, LiveHouse-TS shifts time series benchmarking from snapshot accuracy to continuous temporal validity. Rather than acting as a one-off leaderboard, our infrastructure serves as a continuous time series infrastructure designed to explore vital, long-term scientific questions: Can model rankings be maintained over the long term? Which models remain genuinely robust under distribution shifts? Extensive streaming evaluations across 11 domains with 17 datasets demonstrate that static rankings undergo a dramatic reshuffling under a live protocol.
1 Introduction
LiveHouse-TS addresses the limits of static, frozen-window TSFM benchmarks by evaluating forecasts prequentially on continuously arriving real-world data. It reframes benchmarking around continuous temporal validity, enabling investigation of long-term ranking stability and robustness under distribution shifts.
- TSFMs enable promising cross-domain zero-shot forecasting through large-scale pretraining and zero-shot inference.
- Static benchmarks use predetermined train and test windows, so snapshot leaderboards can miss changing operational conditions.
- Model performance and rankings can change over rolling real-world timelines, particularly when seasonal shifts cause one model to degrade while another remains reliable.
- LiveHouse-TS introduces a leakage-resistant open-world infrastructure with continuous streaming observations and an extensible registry for expanding data sources.
- Its strict prequential protocol requires predictions at current world time before ground truth exists, while evaluation and rankings are updated as new data arrives.
- LiveHouse-TS rankings differ from prior static-benchmark rankings, indicating that strong static performance may not translate to practical deployment.
2 Related Work
Related work spans cross-domain Time Series Foundation Models, language-model adaptation for forecasting, and benchmark infrastructures for standardized and reproducible evaluation. TSFMs use large-scale pretraining for zero-shot or lightly fine-tuned forecasting on unseen datasets, while benchmarks have evolved from fixed competitions and archives toward fairer comparisons.
- Time Series Foundation Models: TSFMs are pretrained on large cross-domain time series corpora and applied zero-shot or with light fine-tuning to unseen datasets.Model families differ in tokenization, architecture, and pretraining objectives.
- Time Series Foundation Models: TSFM architectures include DeepAR, N-BEATS, N-HiTS, PatchTST, DLinear, TimesNet, Informer, Autoformer, FEDformer, Pyraformer, Crossformer, SCINet, and TiDE.The passage presents these as representative model families spanning earlier and later developments.
- Language Models for Forecasting: Another research line reprograms or fine-tunes frozen language models for time series forecasting.This work proceeds in parallel with TSFM development.
- Time Series Forecasting Benchmark: Early forecasting competitions and archives, including M4, M5, and Monash, standardized scoring or consolidated datasets for comparison and pretraining.A later wave focused on fair and reproducible comparison through benchmarks such as TFB.
3 Benchmark Details
LiveHouse-TS is a continuously updated, open-world benchmark that standardizes heterogeneous forecasters, constructs dataset-specific rolling tasks, and evaluates eligible predictions on future observations. Its design emphasizes leakage resistance, fair temporal comparison, scalability, broad dataset diversity, and live-specific stability metrics.
- Architecture: Three coordinated components standardize models, transform public streams into dataset-specific rolling tasks, and evaluate forecasts in a live-house architecture.The components are Model Entrance, Live Data, and Evaluation House.
- Architecture: Model Entrance exposes a unified interface, validates horizon-aligned outputs, and converts heterogeneous forecasts into common scoring representations.It supports hosted TSFMs and lightweight statistical baselines, including mean forecasts for point-error metrics.
- Task construction: Dataset-specific context lengths, horizons, and frequencies ensure identical inputs and targets for models while respecting each stream’s natural timescale.Tasks are windowed after streams are cleaned and mapped into a shared schema.
- Evaluation protocol: A future-only admission filter excludes tasks issued before each model joined, then scores eligible forecasts after corresponding future targets become available.This supports leakage-resistant evaluation and fair comparisons among models entering the leaderboard at different times.
- Data registry: The registry spans 17 datasets, 15 public sources, 11 domains, and 8 native frequencies, combining diverse forecasting regimes with continuously arriving observations.Coverage includes source/domain, temporal-scale, task-structure, and dynamic richness axes, while live collection enables post-participation evaluation.
- Metrics: Evaluation reports RMSE, MAPE, CRPS, Average Rank, Win Rate, and Elo, and adds Temporal Stability to measure performance variation across time.RMSE uses z-normalized series; MAPE applies when targets are bounded away from zero; CRPS is estimated from emitted quantiles.
4 Experiments
Experiments evaluate TSFMs on real future data across multiple domains, comparing live performance with static rankings and examining robustness to temporal drift and ranking changes. Results show that live evaluation can substantially reorder models and reveal evolving performance under deployment-like conditions.
- Zero-shot ability: LiveHouse-TS evaluates zero-shot forecasts across 10 eligible datasets using ranking metrics derived from MAE, MAPE, and CRPS.Per-dataset results are aggregated by domain, sampling frequency, and forecasting horizon.
- Static vs. live rankings: LiveHouse-TS produces a markedly different CRPS ranking from three static benchmarks, whose rankings consistently place Chronos-2 first, followed by TiRex and TimesFM-2.5.Chronos-Bolt and Sundial remain near the bottom of the static rankings, whereas Moirai-2.0 and Toto-1.0 rise in the live comparison.
- Drift robustness: Moirai-2.0 achieves the best average rank on both Temporal Stability and Improvement, indicating stronger robustness to temporal distribution shift than several static top performers.The live benchmark continuously evaluates newly arriving observations rather than a fixed test set.
- Ranking stability: Model rankings continue to evolve across four consecutive weekly snapshots instead of converging to a stable ordering.This dynamic behavior supports continuous live evaluation rather than reliance on a single leaderboard snapshot.
- Qualitative analysis: Forecast visualizations expose shared failure modes under evolving distributions, including oversmoothing, delayed adaptation, and underestimated distribution shifts.On Wave, several models gradually weaken toward constant trajectories rather than preserving periodicity, while Wiki predictions from Moirai-2.0 and Chronos-2 track the ground truth similarly.
5 Conclusion · A Getting Started · A.1 How to join the leaderboard
LiveHouse-TS provides a transparent, reproducible benchmark and live leaderboard for evaluating TSFMs across diverse datasets and forecasting horizons, while highlighting robustness and generalization gaps under shifting conditions. External participants contribute through sustainable hosted forecasting endpoints, validated and admitted only for future live evaluation rounds.
- 5 Conclusion: The benchmark standardizes data processing and evaluation protocols to make TSFM comparisons more transparent and reproducible across diverse datasets and forecasting horizons.The authors identify remaining robustness and generalization gaps when conditions shift across domains.
- 5 Conclusion: LiveHouse-TS uses publicly accessible streams, excludes private or personally identifiable information by design, and reviews data sources for accessibility, licensing, and provenance.Future contributors are expected to exclude sensitive personal data.
- A Getting Started: Participants connect TSFMs by hosting a forecasting endpoint that the leaderboard calls through a standardized API, because live evaluation requires sustainable repeated inference.The leaderboard does not download model weights or execute user code; participants manage inference resources.
- A.1 How to join the leaderboard: Joining requires a public Hugging Face model repository, a public endpoint-implementation URL, and a stable HTTPS forecasting endpoint hosted on participant-controlled infrastructure.A paid Hugging Face Space, participant-owned domain, and public server IP are not required.
- A.1 How to join the leaderboard: The recommended workflow initializes and customizes the portable endpoint, validates local and public routes, publishes metadata and a validation receipt, and requires Python 3, Docker Engine, and Docker Compose v2.Using the default Tailscale route additionally requires Tailscale 1.52 or later on the inference server.
- A.1 How to join the leaderboard: Each request exposes only causal target history, an opaque series identifier, forecast horizon, frequency, and requested quantiles; validation checks health, response validity, finiteness, size, and HTTPS.Future observations, ground truth, raw dataset names, private metric values, and other models’ predictions are withheld.
- A.1 How to join the leaderboard: After validation, submissions are reviewed before activation, and accepted models enter only future live rounds rather than being backfilled into earlier releases.Results appear after the next successful evaluation cycle, while published outputs contain metric summaries and aggregate leaderboard tables.
A.2 How to contribute new dataset
LiveHouse-TS enables community contributions of public time series streams through a standardized live evaluation loop. Contributors define the forecasting problem and provide an adapter for recurring data ingestion, after which the infrastructure automatically forms and scores forecasting tasks.
- Contribution workflow: Contributors register dataset semantics and provide an ingestion adapter so new public time series streams enter LiveHouse-TS’s recurring live evaluation loop.The evaluation house automatically forms and scores forecasting tasks from freshly ingested observations.
B Implementation Details of LiveHouse-TS … C.3 Event-derived series
LiveHouse-TS implements a unified, auditable live-benchmark pipeline that standardizes model forecasts, ingests provenance-preserving data, enforces leakage-resistant task construction, and evaluates models with shared accuracy, probabilistic, and ranking metrics. Its dataset pipeline also converts event streams into regular time series so they follow the same forecasting protocol.
- B.1 Model Entrance.: The model entrance admits hosted, external, local, and statistical methods through a uniform forecasting contract, insulating downstream evaluation from method-specific inference logic.Forecast outputs are validated for shape, horizon length, and finite values before conversion into the common evaluation format.
- B.2 Live Data.: LiveHouse-TS ingests public streams into provenance-rich observations and generates standardized tasks with timestamps, availability times, ingestion times, horizons, covariates, and frequencies.Contributed sources reuse the same live loop, while raw responses are archived without overwriting prior evidence.
- B.3 Evaluation House.: The evaluation house applies identical target windows, metric implementations, and aggregation rules, while storing fine-grained model–dataset–release records and equal-weighted dataset-level live scores.It also supports grouped analyses by domain, frequency, and prediction length, plus matched-comparison ranking and audit metadata.
- B.4 Evaluation Metrics: Evaluation combines lower-is-better point metrics with CRPS, which assesses probabilistic forecasts from model-emitted quantiles by rewarding calibration and sharpness.CRPS is averaged across forecast coordinates and steps and reduces to absolute error for point-mass forecasts.
- B.4 Evaluation Metrics: Relative standing is summarized through Average Rank, Win Rate, and Elo, respectively capturing scale-free dataset ranks, pairwise victories, and strength-weighted head-to-head outcomes.These metrics provide complementary higher- or lower-is-better views of consistency and competitive performance.
- C Dataset details: The dataset appendix documents monitoring entities, forecasting targets, public endpoints, and background notes, with all acquisition sources openly accessible without proprietary API keys.Table 4 supports verification of dataset diversity and provenance.
- C.1 Data pipeline: The data pipeline archives raw responses, parses them into relational metadata and long-format observations, and exports model contexts separately from evaluator-only future targets.Task generation preserves source, entity, variable, timestamp, availability, ingestion, value, frequency, unit, and raw-record provenance while enforcing pre-issuance availability.
- C.3 Event-derived series: GDELT news volume and USGS earthquake events are aggregated into regular time buckets before task construction, allowing event-derived streams to use the standard leaderboard protocol.GDELT provides 15-minute climate-change news-volume data, while USGS supplies a rolling one-week global earthquake GeoJSON feed.
C.4 Diversity verification samples
Before live deployment, LiveHouse-TS verified that all 17 datasets across 11 domains could be downloaded and converted into regular numeric time series. The verification comprised 2,672 representative observations and produced descriptive registry statistics from canonical pipeline rows rather than fixed live train/test splits.
- Verification coverage: 17 datasets across 11 domains yielded usable regular time series in pre-deployment verification, covering 2,672 representative observations.Every source listed in Table 4 passed the download-and-conversion check.
- Verification coverage: The reported per-dataset windows and descriptive statistics are verification slices, not fixed train/test sizes for live evaluation.Table 5 records registry windows and descriptive statistics for the verification samples.
- Statistical construction: Statistics were computed from canonical pipeline rows after selecting a target and representative entity or aggregate, deduplicating observations, and calculating N, mean, range, and standard deviation.The diversity-report script applies these operations systematically rather than relying on manual measurements.
C.5 Dataset inventory and verification statistics · C.6 Per-dataset verification charts
The benchmark registry combines diverse temporal behaviors, while verification charts use short public samples to confirm that each source can be parsed into regular numeric series. The inventory documents dataset access details and abbreviations used throughout subsequent tables and figures.
- C.5 Dataset inventory and verification statistics: The registry mixes smooth environmental signals, volatile financial series, sparse attention counts, and event-driven streams.These behavioral patterns motivate combining heterogeneous data sources under one benchmark.
- C.5 Dataset inventory and verification statistics: Table 4 records background and public access information for each benchmark dataset.Endpoints are listed as base collection URLs, while query parameters vary by entity, time window, and variables.
- C.5 Dataset inventory and verification statistics: Covariates may be omitted in univariate runs.The inventory also defines abbreviations used in subsequent tables and figures, including BTC, PM2.5, Quake, Potomac, Water, and Wave.
- C.5 Dataset inventory and verification statistics: Table 6 summarizes representative dynamics observed in diversity verification samples from June 2026.The table is explicitly presented as a summary of the behavioral patterns seen in the verification charts.
- C.6 Per-dataset verification charts: Figure 12 shows verification-slice target trajectories for each registry dataset.The panels collectively illustrate the diversity of observed temporal behavior summarized in Table 6.
- C.6 Per-dataset verification charts: The verification charts are short public samples used to check that each source can be parsed into a regular numeric series.They are explicitly not fixed test windows.
- C.6 Per-dataset verification charts: Each chart panel corresponds to one registry dataset.For Citi Bike, each point averages available bikes across 50 stations before connecting the sequence.
C.7 Data attribution and usage
LiveHouse-TS acknowledges a broad set of public data providers and summarizes dataset registry and verification statistics. Live deployments should follow providers’ terms, attribution requirements, and request-rate limits.
- Data providers: Public data providers include Open-Meteo, NASA POWER, USGS, NOAA, GBFS/Citi Bike, Binance, CoinGecko, Wikimedia Foundation, World Bank Open Data, and GDELT.These providers are summarized in Table 4.
- Dataset registry and verification: Table 5 reports each dataset’s native and recommended evaluation frequencies, history and forecast horizons, time-series type, row count, and target-variable statistics.The verification slice’s N is a row_count rather than a fixed train/test size; mean, standard deviation, and range summarize the selected target variable.
- Verification variability: Table 7 reports verification variability for the dynamic-richness criterion using CV computed as std/|mean| on the selected target variable.The short verification slices illustrate temporal behavior but are not fixed train/test windows.
- Usage requirements: Live deployments should respect each provider’s terms of use, attribution requirements, and request-rate limits.These operational requirements apply to users operating the benchmark in a live deployment.
D Detailed Forecasting results … E.3 Release-level pairwise score
The online benchmark reports detailed per-dataset forecasting performance and evaluates historical model comparisons using eligible shared releases, unified scoring, and pairwise MSE–CRPS outcomes. These procedures account for asynchronous admission and support direct comparison of point and probabilistic forecasts.
- D Detailed Forecasting results: Tables 8–10 report per-dataset RMSE, MAPE, and CRPS for every model in the frozen online benchmark snapshot.Tables 11–12 additionally report per-dataset Temporal Stability and Improvement from release-level MSE histories.
- D Detailed Forecasting results: Overall, domain-level, frequency-level, and forecast-horizon-level tables aggregate zero-shot rankings using Average Rank, Win Rate, and Elo.Overall standings aggregate per-dataset values, while domain and frequency rankings aggregate within their respective groups; horizon rankings average dataset ranks for each prediction length.
- E Pairwise Historical Ranking: Models admitted at different times are compared only on releases completed in common, avoiding misleading comparisons caused by unequal exposure to historical conditions.Early models may experience volatile and calm periods, whereas later models may be evaluated only under recent conditions.
- E Pairwise Historical Ranking: A release is a resolved causal forecasting task in which admitted models share the same context, sampling frequency, and forecast horizon before observing the target window.Forecasts are frozen before the future target is available, and scoring occurs once the complete target window has been observed.
- E.1 Shared releases and pair eligibility: Pairs are officially eligible only with at least 30 shared releases, 5 datasets, and 7 days of temporal coverage; otherwise comparisons are unavailable rather than ties.Model E had 12 shared releases from 3 datasets over 2 days and therefore lacked sufficient overlap for official comparison.
- E.2 Comparing point and probabilistic forecasts: Point and probabilistic models are evaluated with MSE and CRPS under the same context-only normalization, with point forecasts represented as degenerate predictive distributions.For point-only models, CRPS reduces to normalized MAE; native quantiles are used when available, and output handling is fixed before target observation.
- E.3 Release-level pairwise score: Release-level pairwise scores equally weight lower-is-better MSE and CRPS, assigning 1 for wins on both, 0 for losses on both, and intermediate values for split or tied outcomes.The same rule applies to point–point, probabilistic–probabilistic, and point–probabilistic comparisons; MSE measures central accuracy and CRPS compares predictive distributions.
E.4 Dataset-balanced pairwise win rate · E.5 Model-level ranking and status
LiveHouse-TS balances pairwise comparisons across datasets and eligible opponents so high-frequency streams and long shared histories cannot dominate rankings. It assigns official ranks only to sufficiently connected models and reports evidence fields distinguishing established from provisional results.
- E.4 Dataset-balanced pairwise win rate: Scores are averaged within each shared dataset before averaging across datasets, preventing high-frequency streams from dominating the ranking.Every shared dataset receives equal weight regardless of its release count.
- E.4 Dataset-balanced pairwise win rate: A dataset-balanced win rate above 0.5 means model i wins more often than model j after dataset balancing, but it is not a significance test.For example, W_AB = 0.62 and W_BA = 0.38.
- E.4 Dataset-balanced pairwise win rate: Chronos-2 achieves the highest overall dataset-balanced pairwise win rate, followed closely by TiRex, with higher values indicating better performance.The overall score aggregates eligible metric-specific comparisons, historical releases, and opponents.
- E.4 Dataset-balanced pairwise win rate: TimesFM-2.5 achieves the highest dataset-balanced pairwise win rate in the MSE-based point-forecast track.The ranking covers eligible historical releases and opponents.
- E.4 Dataset-balanced pairwise win rate: Chronos-2 achieves the highest CRPS-based score among the eight TSFMs whose model families natively support probabilistic forecasting.Support is provided through quantiles or forecast samples.
- E.5 Model-level ranking and status: The model-level score is a macro-average over eligible opponents, giving each opponent equal weight rather than allowing longer shared histories to dominate.Models are ordered by decreasing W_i(T).
- E.5 Model-level ranking and status: Official ranks require at least three eligible opponents and membership in the main connected comparison-graph component; otherwise, models remain Provisional.Missing pairs are omitted rather than imputed as 0.5, so some models have neither scores nor ranks.
- E.5 Model-level ranking and status: The leaderboard reports eligible opponents, shared releases, shared datasets, and covered time span to expose evidence supporting each result.These fields distinguish established rankings from provisional ones.
F Overall Model Ranking
The latest leaderboard snapshot presents eligibility-aware overall, point-forecast MSE, and probabilistic CRPS rankings, which do not produce identical model orderings. Chronos-2 leads the overall and CRPS views, TimesFM-2.5 leads MSE, and TiRex remains consistently competitive.
- Overall aggregation: The eligibility-aware overall aggregation ranks Chronos-2 first (0.671), narrowly ahead of TiRex (0.668) and TimesFM-2.5 (0.609).Scores are dataset-balanced pairwise win rates averaged equally over eligible opponents and historical releases; higher values indicate better relative performance.
- Point-forecast MSE: TimesFM-2.5 leads the point-forecast MSE track (0.680), followed by TiRex (0.673) and Chronos-2 (0.668).This view separates point accuracy from distributional forecast quality.
- Probabilistic CRPS: Among eight TSFMs natively supporting probabilistic forecasts, Chronos-2 leads CRPS (0.634), followed by TiRex (0.615) and Moirai-2.0 (0.546).The probabilistic comparison uses quantile- or sample-based forecasts.
- Cross-track interpretation: MSE and CRPS rankings use different eligible model sets and opponent groups, so their absolute scores should be interpreted within each figure rather than compared directly across tracks.The differing orderings show that point-forecast accuracy and probabilistic forecast quality need not align, while TiRex remains competitive across all three views.