What Does an Observability Foundation Model Know?

1 Introduction

• A probe on hidden states may recover a distinction already visible in the input, and even improved recoverability does not establish that the forecaster uses it.A probe score alone cannot distinguish these possibilities.Source: §1 · p. 1

Contributions

• The audit pairs held-out probes with input and backbone baselines, tests forecast response with donor exchange, and evaluates unchanged probe transfer to external datasets.Backbone comparisons use random weights or trained blocks in shuffled order.Source: §1 · p. 1, Contributions · p. 2
• Cadence and metric type beat every tested raw-window model in all five resplits; domain is nearly tied, while cardinality is recovered far better from the raw window.MOMENT-base shows related cadence, metric-type, and domain readouts; residual exchanges move targeted readouts without consistently moving forecasts.Source: Contributions · p. 2

Forecasting foundation models and observability

• Chronos and TimesFM use autoregressive prediction, while MOMENT uses masked reconstruction; Toto adapts the forecasting-foundation-model paradigm to multivariate telemetry.BOOM supplies real observability series and forecast tasks.Source: Forecasting foundation models and… · p. 2

Probing, selectivity, and controls

• Prior probing work studies properties recoverable from representations, while probe capacity and control tasks help distinguish representation-specific structure from accessible structure.Recent time-series work probes temporal concepts and factors [Wiliński et al., 2025, Pandey et al., 2025].Source: Probing, selectivity, and controls · p. 2

Decodability, interventions, and model use

• A readable direction need not be used for prediction, so the paper adopts a donor exchange to test whether moving a readout moves a forecast.Related work uses interventions to connect internal changes with behavior [Hewitt and Liang, 2019, Meng et al., 2022, Turner et al., 2024, Geiger et al., 2024, 2025].Source: Decodability, interventions, and model… · p. 3

Forecast evaluation and transfer

• The external test applies a BOOM-trained probe unchanged to external FEV configurations and long-sequence datasets, rather than testing forecast or representation transfer broadly.The paper distinguishes this narrower test from broader evaluation suites [Shchur et al., 2025, Zhou et al., 2021, THUML, 2024].Source: Forecast evaluation and transfer · p. 3

Models

• Toto’s public checkpoint is read at its patch embedding and layers 0–11; MOMENT-base is read at layers 3, 6, 9, and 11 using a different context and patch length.Toto uses a 1024-observation context with 64-observation patches; MOMENT uses 512 observations with 8-observation patches.Source: Models · p. 3

Data

• BOOM contains 2,807 series and approximately 350 million observations; five seeded, series-disjoint resplits use 500 training, 421 validation, and 422 test series each.The fixed context-and-target length excludes 489 series, including daily and weekly cadences; external transfer covers 11 FEV configurations and four long-sequence datasets.Source: Data · p. 3

Pretraining/evaluation overlap

• Toto was pretrained on an internal observability corpus, and its overlap with BOOM is unknown, although the resplits prevent within-study train/test leakage.The source of any signal therefore remains unattributed.Source: Pretraining/evaluation overlap · p. 3

Labels

• The taxonomy probes cover cadence, metric type, domain, and cardinality; other and multilabel metric or domain entries are excluded from singleton-label probes.Cadence is restricted to Short and Medium; cardinality uses four variate-count bins.Source: Labels · p. 3
• Dynamic labels use context and next-patch statistics: burstiness tracks the largest scaled deviation, sparsity the exact-zero fraction, and coordination the dominant direction of active variate shifts.Coordination is zero when no variate is active.Source: Labels · p. 4

Held-out readouts

• Held-out readouts use logistic-regression probes for categorical labels and ridge probes for continuous labels across 39 layer-and-token views.Views are selected on validation performance; cardinality uses macro-F1 for selection.Source: Held-out readouts · p. 4

Baselines

• Input baselines include six last-patch statistics, four raw-window models, and Toto with random weights or permuted pretrained blocks.The raw-window models are trained from scratch, unlike the linear probe on a representation pretrained elsewhere; a gap does not show that the input lacks the information.Source: Baselines · pp. 4–6
• Toto is higher for cadence and metric type in all five resplits against the strongest raw-window model.Table 1 reports held-out test macro-F1 and paired gaps over five resplits.Source: Baselines · p. 5, Table 1 · p. 5

Common-support probes

• Common-support probes retest gaps over the six-statistic probe and random initialization within fixed strata of the other two labels.They balance target classes by series and do not refit raw-window models within strata.Source: Common-support probes · p. 6

Donor exchange

• Donor exchange blends residuals from high-burst or randomized donors into low-burst target states and checks both the readout and forecast response.The future-burstiness probe is fixed at layer 11 with all-context, series-mean pooling; the primary endpoint compares forecast burstiness and the secondary endpoint is the WAPE ratio.Source: Donor exchange · p. 6

Reporting conventions

• A resplit is the analysis unit, and the reported half-width describes variability across five resplits of one corpus rather than a significance test.A win means a positive paired gap in a resplit, so 5/5 denotes consistency of direction within BOOM.Source: Reporting conventions · p. 6

Zero-shot probe transfer

• The zero-shot transfer test applies the BOOM-trained coordination probe unchanged to Toto activations on 11 FEV configurations and four LSTF datasets.It tests carryover of the fitted readout, not forecast accuracy; whether external representations encode coordination is a separate refit question.Source: Zero-shot probe transfer · p. 6

5.1 Cadence and metric type are more recoverable from Toto than from input baselines

• Cadence and metric type beat every tested baseline in all five resplits, with cadence showing the larger margin and metric type a smaller but consistent one.Both also beat random initialization and block permutation in 5/5 resplits.Source: §5.1 · p. 7
• Domain is nearly tied with raw-window models, while neural raw-window models reach cardinality macro-F1 near 0.95, far above Toto’s 0.471.For cardinality, the six-statistic probe and GBDT also exceed Toto (0.635 and 0.663 vs. 0.471); zero-padded variate channels and coverage expose variate count directly.Source: §5.1 · p. 7

5.2 Common-support probes preserve two gaps in tested strata

• Within fixed strata of the other two labels, cadence and metric type retain positive gaps over the six-statistic probe and random initialization in every resplit.Domain gaps are smaller and less consistent; domain is also most associated with metric type (series-level Cramér’s V = 0.402 ± 0.011).Source: §5.2 · p. 7, Table 2 · p. 8

5.3 MOMENT shows related taxonomy readouts on BOOM

• MOMENT-base’s cadence, metric-type, and domain readouts beat random initialization and the six-statistic probe in all five resplits, while its cardinality readout matches random initialization and trails the six-statistic probe.Its baselines exclude Toto’s raw-window models and block permutation, and the cross-model recurrence does not rank architectures.Source: §5.3 · p. 7, Table 3 · p. 8

5.4 Exchanges move the readout without consistently moving the forecast

• In Toto, donor exchange moves the future-burstiness readout as intended, but forecasts are not consistently burstier under high-burst than randomized donors across blends.MOMENT’s probe check also exceeds parity while lower-future-MAE responses average below parity; its null donors are matched on variate count, last-patch statistics, and taxonomy, unlike Toto’s randomized donors. The figure keeps the probe check fixed across blends and plots forecast endpoints against parity; exchanges measure relative response, not natural causal use.Source: §5.4 · pp. 7–8, Figure 4 · p. 8

5.5 Zero-shot coordination-probe transfer is negative

• Applied without refitting, the BOOM-trained coordination probe has negative R2 on both external suites, doing worse than mean prediction by a larger margin on LSTF than FEV.This tests zero-shot probe transfer, not whether external-series representations contain coordination information; that would require refitting on those series.Source: §5.5 · p. 8

A label-by-label profile

• Cadence has the largest advantage over its strongest raw-window model, metric type a smaller but consistent one, domain is nearly tied, and cardinality is far better recovered from the raw window.These comparisons describe linear recoverability for the evaluated representation, probe family, and baselines, not all information in the model.Source: A label-by-label profile · p. 9

Why cardinality differs

• Toto’s selected cardinality view averages variates without explicit count, while neural raw-window models see the count directly; mean pooling is only one candidate explanation.Architecture, input format, and readout capacity also differ.Source: Why cardinality differs · p. 9

What cadence may reflect

• The Short/Medium cadence metadata may track temporal regularities visible across Toto’s patch context, but this hypothesis is left to controlled tests.Source: What cadence may reflect · p. 9

Readout is not use

• Exchanging residuals moves the future-burstiness readout as intended in both models, but forecasts do not respond consistently; this result is specific to the exchange and endpoint.The proposed next Toto test would match donors on context and taxonomy, preserve target context, include sham exchanges, and report several forecast endpoints.Source: Readout is not use · p. 9

Limits and next steps

• Five resplits of one corpus measure split variability rather than uncertainty over independent observability data, and unknown pretraining overlap leaves the signal’s source unattributed.The cadence label covers only Short and Medium series, and common-support probes omit raw-window models.Source: Limits and next steps · p. 9

Recommendations for practice

• Interpret each probe score against the strongest baseline it beats, including an architecture without its trained configuration, and separately test whether moving the readout moves forecasts.Cadence and metric type pass the first two comparisons here, while the future-burstiness forecast test shows no consistent change.Source: Recommendations for practice · p. 9

Broader impacts

• Treating a probe score as a steering target could produce unintended forecast changes, and the donor-exchange readout is not a validated forecast-control knob.Reporting representation and forecast outcomes side by side keeps that boundary visible.Source: Broader impacts · p. 9

7 Conclusion

• Across five series-disjoint BOOM resplits, cadence and metric type exceed every tested raw-window model, domain is nearly tied with its strongest raw-window model, and cardinality is far better recovered from the raw window.MOMENT-base also shows cadence, metric-type, and domain readouts above random initialization and the six-statistic probe; exchanges move the targeted readout without consistent forecast change, and the BOOM-trained coordination probe has negative zero-shot R2 on tested external benchmarks.Source: §7 · p. 10
• Code and audited results are publicly available.Source: §7 · p. 10

Data and splits

• Each seeded resplit partitions 2,807 BOOM series 70/15/15, caps training at 500 series, and yields 500/421/422 train/validation/test series.Up to four windows per series are sampled; contexts contain 1,024 observations with 64-observation patches and the target is the next patch.Source: Data and splits · p. 13

Views and probes

• Probe views span 13 layers and three token positions with series-mean pooling, while block-permutation probes also allow per-variate pooling; validation selects the view.Categorical probes use logistic regression and continuous probes use ridge regression, with no regularization search.Source: Views and probes · p. 13

Baselines

• Raw-window neural models use masked contexts and a coverage channel, while the GBDT uses 15 engineered features; model training and selection are specified separately.The six-statistic probe, shuffled-label probe, random initialization, and block permutation provide additional controls.Source: Baselines · p. 13, Table 4 · p. 13

Donor exchanges

• Toto’s donor exchange compares low-future-burstiness target windows with high-burst donors from other series and randomized donors, without covariate or taxonomy matching.MOMENT’s matched interchange uses donors selected by future burstiness and variate count, with context-matched null donors; its endpoint is future MAE.Source: Donor exchanges · pp. 13–14

Zero-shot probe transfer

• The BOOM-trained continuous-label probe is applied without refitting to external FEV configurations and LSTF datasets, with transfer averaged equally across datasets within each resplit.The external evaluation covers 11 FEV configurations and four LSTF datasets.Source: Zero-shot probe transfer · p. 14

Compute

• The main five-resplit suite used 16 CPUs for about 1:07–1:25, while GPU stages took 1–37 minutes depending on the experiment.Peak memory was 18.7–21.3 GB for the CPU suite, 28.6–29.1 GB for MOMENT activations, 6.7–9.0 GB for Toto donor exchange, and 1.7–3.0 GB for MOMENT matched interchange.Source: Compute · p. 14

B Within-BOOM Held-Out-Combination Stress Tests

• Rotated tests withhold support-valid domain or taxonomy combinations from training and validation, then evaluate their matching test subsets.Different values are withheld across resplits, so the Table 5 brackets are coverage ranges, not confidence intervals.Source: B Within-BOOM Held-Out-Combination… · p. 14, Table 5 · p. 14
• The rotated tests characterize within-BOOM coverage under the tested support, not generalization beyond that setting.Source: B Within-BOOM Held-Out-Combination… · p. 14

C Additional Diagnostic Tables

• The Toto donor-exchange table reports the fraction of 40 targets with burstier forecasts under high-burst versus randomized donors and the median WAPE ratio.Donors come from other series and are not matched on covariates or taxonomy; the probe check is 0.655 ± 0.136 at every blend.Source: Table 6 · p. 15
• The zero-shot coordination-probe table reports macro-averaged R2 by dataset set and resplit for a BOOM-trained probe without refitting.Source: C Additional Diagnostic Tables · p. 14, Table 7 · p. 15
• MOMENT’s coordination readout exceeds the six-statistic probe in 3/5 resplits, while future burstiness does so in 2/5; the six-statistic probe is highly variable.The cross-model readout claim covers cadence, metric type, and domain.Source: C Additional Diagnostic Tables · p. 15
• MOMENT’s matched-interchange probe check is 0.590 ± 0.071, above 0.5 in 5/5 resplits, while the lower future-MAE fraction is the forecast endpoint.The same 40 held-out triples are reused at each blend.Source: Table 9 · p. 15, C Additional Diagnostic Tables · p. 15

D All Raw-Window and Backbone Baselines

• Table 10 lists all neural and tree raw-window models alongside random initialization and block permutation; Table 1 identifies each label’s strongest raw-window model.Table 10 reports held-out test macro-F1 across five resplits, with means and half-widths.Source: D All Raw-Window and Backbone Baselines · p. 16, Table 10 · p. 16

• The paper scopes its claims to five resplits of one BOOM corpus with unknown pretraining overlap, and notes narrower baselines for common-support and MOMENT comparisons.It also reports readout–forecast dissociation and negative zero-shot probe transfer.Source: NeurIPS Paper Checklist · p. 16
• The authors state that model checkpoints, data sources, resplits, probe settings, baselines, donor exchanges, and transfer protocol are specified for reproducing the main results.They report publicly available code, audited result files, a CPU verifier, and setup and GPU rerun instructions; dataset files and model weights are not included in the repository.Source: NeurIPS Paper Checklist · pp. 17–18
• The paper reports Student-t(4) half-widths across five resplits of one BOOM corpus, not significance tests or independent-corpus intervals.Donor triples are reused across blends.Source: NeurIPS Paper Checklist · p. 19
• The paper reports no human-subject research and states that it introduces no new forecasting model or dataset.It evaluates published models and does not present its exchanges as deployment controls.Source: NeurIPS Paper Checklist · pp. 19–20
• The checklist says the paper discusses that treating a probe score as a steering target could produce unintended forecast changes.The donor-exchange readout is not a validated forecast-control knob.Source: NeurIPS Paper Checklist · p. 20
• The checklist reports that all assets and their sources are cited, lists licenses, and states that no dataset or model weights are redistributed.Source: NeurIPS Paper Checklist · p. 21