Source-linked AI summary
This Time is Different: An Observability Perspective on Time Series Foundation Models
Ben Cohen, Emaad Khwaja, Youssef Doubli, Salahidine Lemaachi, Chris Lettieri, Charles Masson, Hugo Miccinilli, Elise Ramé, Qiqi Ren, Afshin Rostamizadeh, Jean Ogier du Terrail, Anna-Monica Toon, Kan Wang, Stephan Xie, Zongzhe Xu, Viktoriya Zhukova, David Asker, Ameet Talwalkar, Othmane Abou-Amal
TL;DR
Observability forecasting must handle diverse, high-dimensional metrics with complex distributions. The paper introduces TOTO, an observability-focused zero-shot forecasting model, and BOOM, a real-world observability benchmark; TOTO achieves state-of-the-art results across BOOM and general-purpose benchmarks.
Problem
Observability metrics are diverse, high-dimensional, and distributionally complex, making accurate forecasting a challenging modeling task.
Source: §1 · p. 1Method
The paper develops TOTO, a zero-shot forecasting foundation model with observability-focused architectural components, and introduces the BOOM benchmark of real-world observability series.
Source: §1 · p. 2Results
TOTO achieves state-of-the-art performance on BOOM and general-purpose benchmarks, improving CRPS by 12% over the next-best method on BOOM.
Source: §1 · p. 3Takeaways & Limitations
BOOM provides a large-scale evaluation environment for real-world observability data, with 350 million observations across 2,807 time series.
Source: §1 · p. 2TOTO and other time-series foundation models assume fixed-interval series, and TOTO currently handles missing points with heuristic imputations.
Source: §6 · p. 10
Abstract
from arXiv · showhide
We introduce Toto, a time series forecasting foundation model with 151 million parameters. Toto uses a modern decoder-only architecture coupled with architectural innovations designed to account for specific challenges found in multivariate observability time series data. Toto's pre-training corpus is a mixture of observability data, open datasets, and synthetic data, and is 4-10$\times$ larger than those of leading time series foundation models. Additionally, we introduce BOOM, a large-scale benchmark consisting of 350 million observations across 2,807 real-world time series. For both Toto and BOOM, we source observability data exclusively from Datadog's own telemetry and internal observability metrics. Extensive evaluations demonstrate that Toto achieves state-of-the-art performance on both BOOM and on established general purpose time series forecasting benchmarks. Toto's model weights, inference code, and evaluation scripts, as well as BOOM's data and evaluation code, are all available as open source under the Apache 2.0 License available at https://huggingface.co/Datadog/Toto-Open-Base-1.0 and https://github.com/DataDog/toto.
1 Introduction
Observability metrics pose scale and modeling challenges that general-purpose forecasting models struggle to address. The paper introduces TOTO and BOOM, and reports state-of-the-art forecasting performance across observability and general-purpose benchmarks.
1 Introduction
- Observability metrics span diverse operational signals and pose challenges through their diversity, high dimensionality, complex distributions, and millions to billions of distinct series.The introduction says these operational conditions make per-series fine-tuning or supervised training of complex models infeasible, while existing general-purpose foundation models struggle to generalize to observability data [10] [11] [12].Source: §1 · pp. 1–2
- BOOM contributes a benchmark of 350 million observations across 2,807 real-world multivariate time series, approximately twice GIFT-Eval’s size.The paper says BOOM uses real-world observability data and provides its data and evaluation code under Apache 2.0.Source: §1 · pp. 2–3
- TOTO is an open-weights, zero-shot forecasting foundation model with observability-focused scaling, attention, and Student-T mixture prediction components.Its training mixture combines observability metrics, public datasets, and synthetic data, with 4–10× more unique data points than other time-series foundation models.Source: §1 · p. 2
- TOTO achieves state-of-the-art performance on observability and general-purpose forecasting benchmarks, with a 12% CRPS improvement over the next-best method on BOOM.The reported general-purpose benchmarks are GIFT-Eval and LSF.Source: §1 · p. 3
2 Related Work
Prior forecasting approaches face scalability and benchmark-coverage constraints, while existing observability studies cover smaller or narrower datasets. TOTO and BOOM address these contexts, and the paper also outlines TOTO’s patch-based architecture.
2 Related Work
- Classical and neural supervised forecasting approaches require per-dataset training, which the paper describes as operationally infeasible at real-world observability scale.Examples include Holt-Winters, tree-based methods, and (S)ARIMA.Source: §2 · p. 3
- Existing time-series foundation models offer zero-shot forecasting on general-purpose benchmarks, and the paper reports that TOTO outperforms them on public benchmarks and BOOM.GIFT-Eval [1] offers standardized evaluation protocols and a decontaminated pretraining dataset for measuring zero-shot performance without test-data leakage.Source: §2 · p. 3
- TOTO achieves state-of-the-art results on LSF and GIFT-Eval, while BOOM applies GIFT-Eval protocols to a benchmark focused entirely on real-world observability data.BOOM has approximately twice as many time-series points as GIFT-Eval.Source: §2 · p. 3
- Prior observability studies examine challenges or models on smaller datasets, whereas BOOM and TOTO’s pretraining corpus are more diverse and generally orders of magnitude larger.The paper also characterizes TOTO as the first observability-specialized foundation model that performs competitively on GIFT-Eval and LSF.Source: §2 · p. 3
- The architecture figure depicts patches of size 64 projected to dimension 768, followed by 11 time-wise transformer blocks and one variate-wise block.The model generates probabilistic next-patch predictions through a Student-T mixture model.Source: Figure 2 · p. 4, §2 · p. 4
3 TOTO
TOTO combines a decoder-only forecasting model with components designed for multivariate observability data, and it is trained on a large mixture of observability, public, and synthetic series.
3.1 Model architecture
- TOTO uses a decoder-only next-patch prediction architecture with non-overlapping 64-point patches, context length 4,096, and embedding dimension 768.Its transformer also uses pre-normalization, RMSNorm, SwiGLU, and RoPE with XPOS.Source: §3.1 · p. 4
- Patch-based causal normalization uses current-patch and past data for scaling, addressing the leakage risk of whole-series statistics and the limited adaptiveness of first-patch scaling.The paper reports that the resulting approach predominantly preserves causality and substantially enhances forecasting performance, particularly for highly nonstationary series.Source: §3.1 · pp. 4–5
3.2 Training data
- TOTO’s training corpus is 4× TimesFM 1.0’s, 5× Time-MoE’s, 6.5× Moirai’s, and over 10× Chronos’s by unique time-series points.The paper also presents ablations of four architecture components using changes in held-out observability-data negative log likelihood when each is disabled.Source: Figure 3 · p. 5
3.1 Model architecture
- Its factorized attention assigns each block to one axis, using 11 time-wise blocks followed by one variate-wise block to control the balance of interactions.The paper says this 11:1 ratio was selected through hyperparameter optimization.Source: §3.1 · p. 5
- For more than 30 variates, TOTO consistently has the lowest CUDA forward-pass time among compared time-series foundation models.The result is reported in Appendix D.4.Source: §3.1 · p. 5
- A Student-T mixture head models heavy-tailed outputs, while a composite negative-log-likelihood and robust loss mitigates degenerate solutions and outlier effects.The tuned loss uses a Cauchy robust loss with δ = 0.1 and λ_NLL = 0.57.Source: §3.1 · p. 6
3.2 Training data
- TOTO’s corpus contains 2.36 trillion points, including 1.59 trillion non-repeated and non-synthetic points; 43% of the mixture is anonymous internal observability metrics.The observability data excludes customer data and identifiable metadata; public collections and synthetic data are also included, and overlapping Chronos data is removed to avoid test leakage.Source: §3.2 · p. 6
4 BOOM
BOOM is an open evaluation framework designed for observability forecasting, built from real-world telemetry and characterized by high dimensionality and challenging statistical properties. Its evaluation follows GIFT-Eval closely, with adjustments for constant and spiky metrics.
4.1 Dataset
- BOOM contains approximately 350 million points across 2,807 real-world metric queries, with a median of 60 variates per series and a smaller BOOMLET subset.Its data comes from a staging environment isolated from the production environment used for TOTO training, and excludes customer data.Source: §4.1 · p. 7
4.2 Domain taxonomy
- BOOM spans observability domains including application usage, infrastructure, databases, networking, and security, illustrated by grouped error, CPU-limit, and command-execution series.Labels are assigned using an LLM and then human-verified.Source: §4.2 · p. 7, Figure 4 · p. 7
4.3 Statistical characteristics
- BOOM’s statistical profiles show greater diversity and irregularity than GIFT-Eval and LSF, including higher spectral entropy and more frequent nonstationarity.BOOM also has a larger lower tail in first-lag autocorrelation, a bimodal ARCH-LM distribution with a large peak near zero, and higher flat-spot values.Source: §4.3 · pp. 7–8, Figure 5 · p. 8
- Extreme values across the examined statistics reinforce that BOOM observability series have distinct and challenging distributional characteristics.The analysis also identifies sparsity, spikes, noisiness, anomaly contamination, high cardinality, and extreme dynamic range as challenges.Source: §4.3 · pp. 7–8
4.4 Evaluation protocol
- BOOM evaluation follows GIFT-Eval’s prediction lengths, strides, and data splits, but uses shifted geometric means and a separate approach for some numerically unstable series.The adjustments address mostly constant values, infrequent spikes, and extreme constant subsequences.Source: §4.4 · p. 8
- BOOM contains 350M points versus GIFT-Eval’s 158M and has a median of 60 variates per series versus 1 for GIFT-Eval and 7 for LSF.Its sampling intervals range from 10 seconds to 1 day, compared with 10 seconds to 1 year for GIFT-Eval; BOOMLET has similar statistics.Source: Table 1 · p. 8
5 Experiments
TOTO is evaluated on BOOM, GIFT-Eval, and LSF against zero-shot foundation models, full-shot neural models, and classical baselines. It outperforms competing models on BOOM, leads GIFT-Eval in rank and MASE, and performs strongly in both zero-shot and full-shot LSF evaluations.
Evaluation setup
- The evaluation compares TOTO across BOOM, GIFT-Eval, and LSF with zero-shot foundation models, target-trained neural models, and classical supervised approaches.Full-shot deep learning models were not evaluated on BOOM.Source: §5 · p. 9
BOOM
- On BOOM, TOTO achieves 13.1% lower MASE and 12.4% lower CRPS than MoiraiBase, with a mean CRPS rank of 2.351 versus 4.278.BOOM metrics are normalized by the Seasonal Naive forecast and aggregated across tasks using shifted geometric means.Source: §5 · p. 9, Table 2 · p. 9
GIFT-Eval
- On GIFT-Eval, TOTO ranks first in rank and MASE and second in CRPS among the reported models.As of May 2025, it has an average ranking score of 5.495, MASE of 0.673, and CRPS of 0.437.Source: Table 3 · p. 9, §5 · p. 9
LSF
- On LSF, TOTO has the best results on 8 of 12 reported metrics among zero-shot methods and the lowest average MAE and MSE.In full-shot evaluations, it also leads on 8 of 12 reported metrics and has the lowest average MAE and MSE of all methods.Source: §5 · p. 9, Table 4 · p. 10
6 Conclusion
The paper presents TOTO and BOOM as advances for forecasting observability metrics and reports state-of-the-art results across BOOM, GIFT-Eval, and LSF. Its appendix reports benefits from causal scaling and an attention design intended to retain time and variate interactions while improving scalability.
6 Conclusion
- TOTO achieves state-of-the-art results on BOOM, GIFT-Eval, and LSF, while BOOM captures high-cardinality production telemetry at scale.The conclusion identifies fixed-interval time-series assumptions, heuristic imputation for missing points, absent calendar features, and unstudied extreme prediction lengths as limitations or future-work directions.Source: §6 · p. 10
A.1 Input/output scaling
- Causal scaling leads to dramatic performance improvements over naive global scaling in the reported ablation.The appendix describes a vectorized Welford algorithm for computing causal statistics and stabilizing training against extreme outliers.Source: §A.1 · p. 18
A.2 Attention mechanism
- TOTO’s attention approach targets both time and variate interactions while reducing computational cost and improving scalability.The appendix contrasts this with factorized attention, which is more efficient than full cross-attention but doubles the network’s effective depth.Source: §A.2 · pp. 18–19
A.2.1 Complexity analysis
The appendix describes how Toto reduces attention complexity and produces probabilistic forecasts, then details its loss and hyperparameter-selection procedure.
A.2.1 Complexity analysis
- Factorized time-wise and variate-wise attention is asymptotically smaller than full self-attention, especially as variate count or sequence length grows.Source: §A.2.1 · p. 20
A.3 Probabilistic prediction
- The Student-T mixture model improves both point-prediction and probabilistic forecasting accuracy over a single Student-T distribution, while accommodating complex output distributions.Source: §A.3 · p. 20
A.4 Forecasting
- At inference, Toto samples from the mixture and feeds each draw back into the decoder, producing independent trajectories and quantile intervals limited by sample count.Source: §A.4 · p. 21
A.5 Loss function
- The robust loss significantly improves point-forecasting accuracy without hurting probabilistic predictions; combined with negative log-likelihood, it supports stable convergence.Source: §A.5 · p. 22
A.6 Hyperparameter optimization
- The hyperparameter sweep ran 133 iterations of 50,000 training steps each, minimizing validation MAE on multistep observability forecasts.Source: §A.6 · p. 22
B.1 Observability dataset
The training corpus combines observability metrics queried from Datadog with public time series datasets.
B.1 Observability dataset
- Datadog metrics come from user-generated queries over internal users’ telemetry, not customer data; queries select and aggregate related variates into multivariate series.Source: §B.1 · p. 24, Figure 8 · p. 24
B.2 Public datasets
- The public corpus draws on the GIFT-Eval Pretrain collection and non-overlapping Chronos datasets, spanning multiple domains and sampling frequencies.Source: §B.2 · p. 24
B.3 Synthetic data
The paper supplements its data sources with synthetic series and preprocessing that diversify training examples, then describes BOOM’s preparation and scale.
B.3 Synthetic data
- Synthetic data makes up approximately 33% of training and combines generated trends, ARMA processes, seasonal patterns, and residual distributions.Source: §B.3 · p. 24
B.4 Preprocessing
- Training preprocessing uses padding, masking, and random time offsets; with 14% probability, it also shuffles adjacent variates across Datadog, public, and synthetic sources.Source: §B.4 · p. 25
C.1 Domain taxonomy
- BOOM groups query results into multivariate series and uses model-suggested labels that human annotators review for consistency and accuracy.Source: §C.1 · p. 25
- BOOM preprocessing imputes missing values by metric type, slices series into fixed windows, caps groups at 100 variates, and filters degenerate test cases.Source: §C.1 · pp. 25–26
- The final BOOM benchmark contains 350 million points across 2,807 metric queries, with varied frequencies, series lengths, and variate counts.Source: §C.1 · p. 26
C.2.1 Prediction terms and evaluation windows
The evaluation defines frequency-based forecast horizons, scores point and probabilistic forecasts, and applies specified inference protocols across models.
C.2.1 Prediction terms and evaluation windows
- BOOM’s short-term horizon is assigned by sampling frequency; medium- and long-term horizons extend it by factors of 10 and 15 when they fit the test window.Source: §C.2.1 · pp. 26–27
C.2.2 Evaluation metrics
- The evaluation uses MASE for point forecasts and CRPS for probabilistic forecasts, with both scores normalized by Seasonal Naive performance on the test split.Source: §C.2.2 · pp. 27–28
C.2.3 Inference procedures
- Foundation models share a 2,048-step context; model-specific sampling procedures are used, while univariate-only TimesFM and Chronos are evaluated variate by variate.Source: §C.2.3 · p. 28
C.2.4 Aggregation of results
TOTO leads BOOM across its taxonomy categories, while the evaluation handles zero-error and near-constant series with separate aggregation procedures. The section also describes BOOMLET and presents benchmark and forecast comparisons.
C.2.4 Aggregation of results
- Because zero errors collapse geometric means, normalized MASE and CRPS use shifted geometric means, while low-variability queries use unnormalized metrics and arithmetic aggregation.NaN and infinite CRPS values are imputed with the mean CRPS across remaining series.Source: §C.2.4 · pp. 28–29
- On near-constant series, zero-shot models have extremely low MASE, while TOTO’s CRPS is slightly elevated and its intervals appear overly wide on some series.The authors conjecture this conservatism may reflect large anomalies in pretraining data and could be desirable for downstream anomaly detection.Source: §C.2.4 · p. 29, Table 10 · p. 29
C.3 BOOMLET
- BOOMLET uniformly samples BOOM metric queries while prioritizing queries with more variates, yielding 32 queries, 1,627 variates, and about 23 million observations.The selection is intended to preserve key taxonomy distributions while retaining practical training volume per query.Source: §C.3 · p. 29
D.1 BOOM
- Qualitative BOOM examples contrast TOTO’s smoother seasonal forecasts, sustained phase alignment, and burst-sensitive distributions with competitors’ overfitting, damping, or flat predictions.Figure 12 shows 336-step zero-shot forecasts with ground truth, median forecasts, and 95% prediction intervals.Source: §D.1 · p. 29, Figure 12 · p. 31
- Across term, metric-type, and domain groups, TOTO consistently achieves the lowest CRPS, with strong margins over all baselines.The reported category metrics use shifted geometric means.Source: §D.1 · p. 30, Table 12 · p. 30, Table 13 · p. 30
D.1.1 BOOMLET
- BOOMLET results are presented in Table 11, whose caption identifies it as reporting full results across evaluated models.The supplied passages do not state the BOOMLET table’s comparative outcomes.Source: §D.1 · p. 29, Table 11 · p. 29, §D.1.1 · p. 30
D.2 GIFT-Eval
- GIFT-Eval spans varied frequencies, domains, series lengths, horizons, and seasonality patterns, and reports point accuracy, probabilistic forecast quality, and overall ranking.Figure 12 is identified as comparative BOOM forecasts, rather than a GIFT-Eval result.Source: §D.2 · p. 30
D.3 LSF
On LSF, TOTO performs strongly in zero-shot evaluation and TOTOFT improves results further, though fine-tuning gains vary across datasets. The section also compares computational efficiency as variate counts increase.
D.3 LSF
- In zero-shot LSF evaluation, TOTO has the lowest average MAE and MSE and beats other zero-shot baselines on 8 of 12 evaluation metrics.The authors note LSF datasets are limited in size and diversity, and supervised baselines may already be near the benchmark’s upper bound [1] [29].Source: §D.3 · p. 32
- TOTOFT ranks first in 8 of 12 metric-dataset pairs and records the lowest average MAE (0.314) and MSE (0.284), with state-of-the-art results on 3 of 6 datasets.It outperforms all models on both metrics for ETTm2, Electricity, and Weather.Source: Table 15 · p. 33, §D.3 · p. 34
- Fine-tuning benefits vary: gains are especially strong on ETTm1, ETTm2, and Weather, modest on ETTh1, and not notable on ETTh2.TOTOFT still does not surpass other full-shot models on ETTh1.Source: §D.3 · p. 34
D.4 Computational Efficiency
- TOTO has the lowest FLOPs across variate counts and is lowest or second-lowest in wall time; from 150 variates onward, it beats Moirai on all reported metrics.The comparison profiles single-sample forward passes, and the authors note TOTO uses KV caching while other models do not appear to.Source: Table 18 · p. 37, §D.4 · pp. 37–38
E Ablations
Ablations identify causal scaling and the Student-T mixture output as especially consequential design choices for validation NLL. The impact statement focuses on forecast reliability and describes BOOM’s shared data scope.
E Ablations
- Ablations disable one component at a time against the full model, with variants trained for 75,000 steps and compared by their best held-out validation NLL checkpoints.Validation was evaluated every 5,000 training steps.Source: E Ablations · p. 38
- Removing causal scaling raises NLL by 27.3%, replacing the Student-T mixture with one Student-T raises it by 27.2%, and removing robust loss raises it by 11.1%.Removing variate-wise attention yields a smaller 1.6% increase; comparisons use held-out observability pretraining data.Source: Table 19 · p. 38, E Ablations · p. 38
F Impact statement
- The impact statement prioritizes forecast accuracy and reliability for infrastructure and application performance; BOOM contains numerical observability series with application labels, no other metadata, and no PII.The authors characterize harms from a time-series forecasting model as considerably lower than those of language or image models.Source: F Impact statement · p. 38