PaperlayerView exact revision on arXiv

Source-linked AI summary

Toto 2.0: Time Series Forecasting Enters the Scaling Era

Emaad Khwaja, Chris Lettieri, Gerald Woo, Eden Belouadah, Marc Cenac, Guillaume Jarry, Enguerrand Paquin, Xunyi Zhao, Viktoriya Zhukov, Othmane Abou-Amal, Chenghao Liu, Ameet Talwalkar, David Asker

arXiv:2605.20119v2cs.LGcs.AI

TL;DR

Time series foundation models have lacked evidence of reliable scaling with a single recipe. Toto 2.0 evaluates a five-model family from 4M to 2.5B parameters and finds smooth improvements across BOOM, GIFT-Eval, and TIME, establishing scaling as a practical tool for TSFMs.

  • Problem

    Time series foundation models lacked evidence that one recipe could produce reliable improvements as model size and token budgets increased.

  • Method

    The paper introduces Toto 2.0, five open-weights forecasting models from 4M to 2.5B parameters trained with a single scaling recipe.

  • Results

    Toto 2.0 sets a new state of the art on BOOM, GIFT-Eval, and TIME, with every model size improving on the one below it.

  • Takeaways & Limitations

    Scaling TSFMs with a single recipe is no longer only a research question but a practical tool, while closing the gap with classical baselines remains open.

  • Takeaways & Limitations

    The paper identifies closing the long-horizon gap with classical baselines as an important remaining bottleneck for TSFM research.

Abstract

from arXiv · show

We show that time series foundation models scale: a single training recipe produces reliable forecast-quality improvements from 4M to 2.5B parameters. We release Toto 2.0, a family of five open-weights forecasting models trained under this recipe. The Toto 2.0 family sets a new state of the art on three forecasting benchmarks: BOOM, our observability benchmark; GIFT-Eval, the standard general-purpose benchmark; and the recent contamination-resistant TIME benchmark. This report describes our experimental results and details the design decisions behind Toto 2.0: its architecture and training recipe, training data, and the u-muP hyperparameter transfer pipeline. All five base checkpoints are released under Apache 2.0.

1 Introduction

Toto 2.0 demonstrates that time series foundation models improve predictably with scale, with five open-weight models topping the evaluated forecasting benchmarks. The introduction previews the family’s architecture, data, hyperparameter-transfer pipeline, scaling results, limitations, and releases.

  • Contribution: Toto 2.0 comprises five open-weight models spanning 4m, 22m, 313m, 1B, and 2.5B parameters, with every size improving on the one below it.The family takes the top spots on BOOM, GIFT-Eval, and TIME.
  • Architecture and training recipe: Contiguous patch masking enables single-pass parallel forecasting, while a quantile head and NorMuon improve stability and optimization at scale.These replace autoregressive decoding, Toto 1.0’s Student-T mixture, and AdamW, respectively.
  • Training data: Toto 2.0 pretrains exclusively on Datadog internal observability metrics and synthetic data, with public data entering only during finetuning at 45% of the mix.This setup is intended to test cross-domain generalization more strongly than pretraining directly on public time-series corpora.
  • Hyperparameter transfer pipeline: A structured search tunes hyperparameters on a 10m proxy and transfers one configuration across all five target sizes using u-µP.The pipeline modifies width, depth, and head count while u-µP makes learning dynamics width-independent.
  • Results and scaling behavior: Toto 2.0 reaches state-of-the-art performance on BOOM, GIFT-Eval, and TIME, with every size on or near the Pareto frontier.Finetuned and ensembled variants top the full GIFT-Eval leaderboard, while inference is faster at long horizons and larger models forecast beyond training context on synthetic multi-scale signals.
  • Where TSFMs go next: The paper identifies long-horizon forecasting, data curation, downstream-value evaluation, and multimodality as future bottlenecks and opportunities, and releases all five model weights under Apache 2.0.The distributed training library is also released as dd_unit_scaling.

2 Architecture

Toto 2.0 retains Toto 1.0’s decoder-only patched-transformer backbone with alternating time-axis causal and variate-axis full attention, while introducing contiguous patch masking, a quantile output head, and NorMuon optimization. These changes enable single-pass parallel forecasting, improve numerical stability during scaling, and adapt optimization to pinball loss.

  • Backbone and redesign: Toto 2.0 retains a decoder-only patched transformer with alternating causal time-axis and full variate-axis attention, while replacing several core components.The redesign includes contiguous patch masking, a quantile output head, and NorMuon replacing AdamW.
  • Contiguous patch masking: CPM trains on variable-length contiguous masked spans and fills the forecast horizon with mask tokens for single-pass parallel decoding.At inference, the model predicts multiple future patches together, mitigating error compounding from autoregressive decoding.
  • Contiguous patch masking: Single-pass decoding generally remains stable up to an approximately 768-step horizon on synthetic multi-scale signals, while block decoding is used for the long-horizon study.Block decoding reuses the KV cache and commits forecast blocks round by round to mitigate overall drift.
  • Quantile output head: The quantile output head predicts nine quantiles at levels T = {0.1, 0.2, . . . , 0.9} using pinball loss instead of Toto 1.0’s Student-T mixture.The Student-T mixture became numerically unstable at large activations and diverged as predictions approached zero; predicted quantiles are sorted during inference to prevent crossing.
  • Optimization: NorMuon optimizes internal matrix-shaped parameters, while AdamW remains assigned to input/output projections, biases, and norms.NorMuon’s row normalization reinstates a β2 variance mechanism at the neuron level, addressing pinball loss’s sign-valued gradients.

3 Training data

Toto 2.0 trains exclusively on Datadog internal telemetry and synthetic data, with 3.40–5.04 T training points versus 2.36 T for Toto 1.0. Its data recipe removes public pretraining data, expands and diversifies synthetic data, and rebalances telemetry toward longer sampling intervals.

  • Dataset scale and composition: 5.04 T data points train the 313m, 1B, and 2.5B models, while 3.40 T train the 4m and 22m models, up from 2.36 T in Toto 1.0.The training mix combines Datadog internal telemetry with synthetic data.
  • Dataset scale and composition: Public data is removed from pretraining because hyperparameter sweeps found mixtures excluding it performed best at proxy model scale.Public data remains in the finetuning recipe.
  • Synthetic data: Synthetic data more than doubles through newer generation methods that produce more diverse regimes.Toto 2.0 replaces Toto 1.0’s generic stochastic processes with TempoPFN’s prior-data fitted network framework, whose prior includes nonstationary trends, abrupt changepoints, and long-range dependencies.
  • Telemetry data: Toto 2.0 rebalances Datadog telemetry by parameterizing sampling intervals and overweighting longer intervals instead of skewing heavily toward 10 s intervals.This exposes the model to a more diverse, higher-signal view of the same underlying telemetry.
  • Telemetry data: The real-world data consists exclusively of Datadog internal observability metrics, including CPU utilization, memory usage, request latency, and error rates, with no customer data used.Compared with Toto 1.0, it draws from broader data sources and more recent periods.

4 Hyperparameter transfer pipeline

Toto 2.0 uses unit-scaled u-µP to tune hyperparameters on a 10m-parameter proxy and transfer the resulting configuration across all five target sizes. Four sequential proxy-search rounds selected the architecture, data mixture, optimizer settings, and decay schedule used for the base models.

  • Transfer principle: u-µP makes the optimal learning rate independent of model width, enabling direct transfer from a cheap 10m proxy to all five target sizes.The method combines Maximal Update Parametrization with unit scaling and avoids retuning each target model.
  • Proxy search: 30,000 steps per proxy trial, with dmodel = 256 and a WSD schedule, reduced each configuration run from days to a few hours.The proxy uses L = 12 and h = 4 at the same batch size as the target models.
  • Round 1: Architecture: PerDimScale, last-positioned variate-axis attention, and contiguous-patch masking with cmax = 16 and pmax = 0.4 formed the best architecture configuration.The sweep compared PerDimScale, QK-Norm, or neither, several variate-attention cadences, bias placement, and masking parameters.
  • Round 2: Data mixture: 42.5% Datadog observability data and 57.5% synthetic data formed the optimal mixture, excluding public data.The Datadog share comprised 20% 10-second, 7.5% 60-second, and 15% 5+ minute metrics, and this mixture was used for all base models.
  • Round 3: Optimizer: η = 0.652 for NorMuon and η = 0.012 for AdamW were selected, with 6,000 warmup steps and gradient clipping at 7.0.NorMuon used µ = 0.96, β2 = 0.999, and weight decay = 2 × 10−8; AdamW used β1 = 0.91 and β2 = 0.972.
  • Round 4: Decay schedule: 10,500 decay steps using linear decay won the schedule search and were maintained for every base model.This tail represented 1.7–2.6% of the 400,000- and 600,500-step training budgets.

5 Results

Toto 2.0 achieves state-of-the-art results across BOOM, GIFT-Eval, and TIME, with larger models leading foundation-model comparisons on the latter two benchmarks. The results also examine inference latency and long-horizon stability beyond forecast accuracy.

  • Benchmark results: Toto 2.0 sets a new state of the art on BOOM, GIFT-Eval, and TIME.Every model size leads on BOOM; the three largest sizes lead foundation models on GIFT-Eval, while the same sizes take TIME’s top three spots on every metric.
  • GIFT-Eval results: 2.5B-FT and Toto 2.0 FnF ensemble take the top two spots outright on GIFT-Eval.The three largest Toto 2.0 sizes lead foundation models on the benchmark.
  • TIME results: On TIME, the same larger sizes take the top three spots on every metric, ahead of every evaluated external foundation model.TIME is a contamination-resistant zero-shot benchmark built from fresh datasets to mitigate established benchmark test-set contamination.
  • BOOM results: All five Toto 2.0 sizes outrank every other foundation model on every BOOM metric.The metrics are CRPS rank, CRPS, and MASE, for which lower is better.
  • Beyond accuracy: Every Toto 2.0 size beats Toto 1.0 at long horizons, while larger sizes retain coherent multi-scale structure beyond their training context.These analyses cover inference latency and long-horizon stability, respectively.

5.1 BOOM · 5.2 GIFT-Eval – foundation models · GIFT-Eval

Toto 2.0 scales across forecasting benchmarks, placing every size on BOOM’s Pareto frontier and ranking first among foundation models on GIFT-Eval despite training only on synthetic and observability data.

  • 5.1 BOOM: BOOM evaluates forecasts for observability signals including CPU utilization, memory, request latency, and error rates.These are the signals production monitoring systems care about.
  • 5.1 BOOM: Every Toto 2.0 size lies on BOOM’s Pareto frontier, with no foundation model forecasting better at the same parameter count.The three largest sizes lead BOOM with CRPS ranks of 3.88 (2.5B), 3.96 (1B), and 4.26 (313m).
  • 5.1 BOOM: 7× parameter efficiency separates Toto 2.0 22m at 5.53 from Toto 1.0 at 6.94, whose model has 151m parameters.The 22m model already clears Toto 1.0 on BOOM.
  • 5.2 GIFT-Eval – foundation models: GIFT-Eval covers 97 evaluation tasks formed from dataset, frequency, and prediction-horizon combinations across 23 base datasets.The datasets span energy, retail, weather, and finance domains.
  • 5.2 GIFT-Eval – foundation models: Toto 2.0 ranks first among foundation models on GIFT-Eval despite training only on synthetic and observability data.The three largest sizes score 20.3 (2.5B), 21.1 (1B), and 21.4 (313m) on CRPS rank.
  • GIFT-Eval – foundation models: 20.3 (2.5B), 21.1 (1B), and 21.4 (313m) lead CRPS rank, while the 313m model is 1.7 points ahead of PatchTST-FM r1 at 23.1.Chronos-2 scores 23.5, and 22m scores 26.8 versus Toto 1.0 at 35.1.
  • GIFT-Eval: Figure 6 compares foundation models across CRPS rank, MASE rank, CRPS, and MASE; Toto 2.0 takes the top three CRPS-rank spots, while 2.5B leads MASE rank.Lower is better for all reported metrics.

5.3 GIFT-Eval – finetuned and ensemble models

This section evaluates Toto 2.0 as a foundation for downstream adaptation through finetuning and ensembling on GIFT-Eval. It distinguishes leaderboard finetuning from broader pretraining use and introduces a per-window ensemble over ten foundation models.

  • Section scope: The results assess Toto 2.0 as a strong starting point for downstream adaptation, not as evidence for the zero-shot scaling claim.The experiments cover a finetuned 2.5B model and an ensemble method.
  • Finetuning: GIFT-Eval provides GIFT-Eval Pretrain and official evaluation-dataset train splits, but only the latter qualifies submissions for the leaderboard’s finetuned tier.The pretraining corpus is curated not to overlap with benchmark evaluation datasets.
  • Ensembling: Toto 2.0 FnF selects per-window weights over ten foundation models, combining all five Toto 2.0 sizes with Chronos-2, TimesFM 2.5, TiRex, FlowState, and PatchTST-FM r1.The ensemble addresses differing dataset preferences for short-horizon priors and broad pretraining coverage.
  • Ensembling: Toto 2.0 FnF follows the FFORMA framework for feature-based forecast model averaging.The supplied passage identifies the framework but does not provide further implementation details.

GIFT-Eval

On GIFT-Eval, the Toto 2.0 FnF ensemble ranks first on every metric, while the finetuned 2.5B ranks second on rank metrics and third on raw metrics. Within the ensemble, Toto 2.0 contributes the largest average assigned weight among candidate models.

  • GIFT-Eval: Toto 2.0 FnF ranks first on every GIFT-Eval metric, tied with TSOrchestra on raw CRPS.The ensemble leaderboard includes foundation, finetuned, ensemble, and agentic systems.
  • GIFT-Eval: The finetuned Toto 2.0 2.5B ranks second on rank metrics and third on raw metrics.This establishes strong performance for the single 2.5B model alongside the ensemble result.
  • GIFT-Eval: 39% of the ensemble’s assigned weight goes to the Toto 2.0 family, ahead of Chronos-2 at 32% and the four remaining external models combined.The meta-learner uses softmax weights to represent each candidate’s contribution to predictions.

5.4 TIME

TIME evaluates forecasting on 98 tasks from 50 fresh datasets, using operationally aligned horizons and avoiding legacy datasets likely present in pretraining corpora. Toto 2.0 occupies the top three positions on every TIME metric, led by the 2.5B model on CRPS rank, MASE rank, and CRPS.

  • Benchmark: TIME comprises 98 forecasting tasks from 50 fresh datasets curated through a human-in-the-loop pipeline.Its horizons reflect real-world operational requirements rather than mechanical short, medium, and long categories.
  • Benchmark: TIME excludes legacy datasets such as ETTh1, Electricity, Traffic, and Weather, replacing them with recent data unlikely to have appeared in pretraining.The benchmark was designed to reduce contamination from datasets circulated through TSFM pretraining corpora.
  • Results: Toto 2.0 takes the top three spots on every TIME metric, with the 2.5B model leading on CRPS rank (3.43), MASE rank (3.54), and CRPS (0.532).Chronos-2 is fourth on CRPS rank (4.03), while PatchTST-FM r1 is fifth (5.04).
  • Results: Scaling is not strictly monotonic within Toto 2.0 on TIME: the 313m model leads on MASE and edges out the 1B model on both rank metrics.This shows that the largest model does not lead every TIME comparison within the Toto 2.0 family.

5.5 Inference latency

Toto 2.0 substantially reduces inference latency through single-pass and block decoding, with every model size faster than Toto 1.0 at a 1,024-step horizon and the 313m model matching Chronos-2 latency. Single-pass latency remains flat up to 768 steps, while the 2.5B model remains faster than Chronos-2 at 4,096 steps.

  • Inference latency: A 1,024-step forecast requires up to 16 autoregressive steps for Toto 1.0 but only a single forward pass for single-pass Toto 2.0.Single-pass decoding processes the entire horizon in one forward pass; block decoding instead generates segments conditioned on each previous segment’s median, using KV caching.
  • Inference latency: Single-pass Toto 2.0 latency stays flat up to a 768-point forecast length, and 2.5B remains faster than Chronos-2 at a 4,096-step horizon.The 768-point length was found best on synthetic signals.
  • Inference latency: Every Toto 2.0 size is significantly faster than Toto 1.0 at a 1,024-step forecast horizon.The 313m model runs at roughly the same latency as Chronos-2, which has 120m parameters.

5.6 Long-horizon stability

Toto 2.0’s long-horizon stability was tested beyond its 4,096-step training context on sinusoidal mixtures. Larger models preserved underlying patterns farther, with 1B stable across all tested horizons and 2.5B more accurate still.

  • Motivation: Long-horizon forecasting requires preserving fine-resolution structure, including spikes, transient anomalies, and sub-period dynamics, which downsampling can lose.Benchmark performance near the training context does not fully capture practical long-horizon, fine-resolution forecasting needs.
  • Evaluation: All five Toto 2.0 sizes were evaluated on randomly generated sinusoidal mixtures at 2,048, 4,096, and 8,192 timesteps.The test extends well beyond Toto’s 4,096-step training context and measures behavior beyond the training horizon rather than genuinely novel dynamics.
  • Results: 4m collapses past its training context, 22m degrades by 4,096 steps, and 313m loses structure beyond 4,096 steps.The smaller models progressively fail to maintain stable long-range forecasts as the horizon increases.
  • Results: 1B maintains the underlying pattern across all three horizons, while 2.5B is more accurate still.The comparison included Toto 1.0 and Chronos-2 across the three horizons.

6 Discussion

Toto 2.0 demonstrates reliable scaling across three benchmarks, while the discussion identifies unresolved gaps versus classical methods and future directions in data curation, metrics modeling, and observability multimodality. Long-horizon forecasts still lose structure, including for the 2.5B model at 8,192 steps.

  • Scaling: From 4M to 2.5B parameters, one recipe yields smooth improvements on BOOM, GIFT-Eval, and TIME, with only minor TIME rank-metric inversions.The paper characterizes Toto 2.0 as the first TSFM family where simply increasing model size reliably improves performance.
  • Closing the gap with classical baselines: Classical methods remain stronger on clean extrapolation, appropriate growth of prediction intervals, and predictable out-of-distribution behavior.Foundation models instead capture multivariate interactions, long context, and transfer across domains.
  • Closing the gap with classical baselines: At a forecast horizon of 8,192 steps, even the 2.5B model loses some structure that a properly fitted seasonal model would extrapolate cleanly.The discussion connects this gap to long-horizon forecasting and also notes issues with tail behavior and regime shifts.
  • Improved data curation: TSFM data curation remains ad hoc, unlike language modeling’s emphasis on quality filtering, deduplication, annotation, mixing, and curriculum.The paper frames improved curation as a future research direction after scaling became the central open question.
  • Metrics as a distinct modality: Datadog metrics constitute a distinct modality whose embedded information is lost when compressed into generic time-series representations.The discussion motivates dedicated modeling of the large volume of metrics data collected at Datadog.
  • Multimodality and world models for observability: Future observability models should address diverse distributed-system data, including metrics, traces, and logs, rather than focusing primarily on time series plus text.The discussion positions multimodality as a path toward models that understand distributed-system behavior.
Loading 2605.20119v2…