PaperlayerView exact revision on arXiv

Source-linked AI summary

Do Time-Series Foundation Models Pay Off for Industrial Monitoring? A Cost-Aware Empirical Study

Guan-Hua Wen, Kuan-Yu Chen

arXiv:2608.22968v1cs.LG

TL;DR

Industrial monitoring requires models that fit heterogeneous targets while respecting calibration and resource constraints, but TSFM deployment value remains uncertain against lightweight baselines. The paper conducts a protocol-aware comparison across C-MAPSS, MIMII, and BDG2, finding that TSFMs are task-dependent options rather than default replacements for fitted lightweight models.

  • Problem

    Evidence for TSFM deployment value is limited when monitoring tasks have heterogeneous definitions and target-normal data make competitive lightweight models available.

  • Method

    The study compares classical one-class methods, compact neural models, residual forecasters, and frozen or zero-shot TSFMs across matched C-MAPSS, MIMII, and BDG2 protocols.

  • Results

    Across the evaluated protocols, fitted lightweight models outperform frozen MOMENT-small on C-MAPSS and MIMII, while forecasting TSFMs show limited BDG2 advantages that do not extend consistently across metrics.

  • Takeaways & Limitations

    TSFMs are task-dependent deployment choices rather than default replacements; compact fitted models remain strong references when target-normal data and resource constraints matter.

  • Takeaways & Limitations

    The conclusions are conditional on heterogeneous protocols and the tested TSFM coverage; larger variants, fine-tuning, adapters, or alternative contexts could change results.

Abstract

from arXiv · show

Industrial monitoring models must detect operationally relevant deviations while satisfying target-specific data, calibration, and resource constraints. Time-series foundation models (TSFMs) promise reusable representations and zero-shot forecasts, yet evidence for their deployment value remains mixed when task definitions are heterogeneous and lightweight baselines are competitive. This work presents a protocol-aware empirical assessment across three settings: a C-MAPSS degradation-risk proxy, normal-only training for anomalous-sound detection on MIMII, and BDG2 forecasting-residual diagnostics with synthetic target perturbations. We assess classical one-class methods, compact neural autoencoders, residual forecasters, MOMENT-small, Chronos-T5, and TimesFM 2.5 in terms of anomaly-ranking performance, risk-horizon sensitivity, residual forecasting and perturbation sensitivity, and local implementation cost. Across 100 C-MAPSS engines evaluated out of fold, TCN-AE reaches fold-weighted AUROC/AUPRC 0.9570/0.8960, compared with 0.7310/0.3080 for MOMENT reconstruction; paired engine-cluster bootstrap confidence intervals exclude zero for both differences. Across five matched MIMII pump evaluations, OCSVM also exceeds MOMENT reconstruction in AUROC and AUPRC. On a fixed 12-meter BDG2 panel, TimesFM 2.5 has the lowest aligned forecast error and the highest synthetic AUROC point estimate, although synthetic AUPRC is similar across TSFM and fitted residual models. Same-device measurements show that MOMENT incurs higher latency, peak allocated VRAM, and serialized state-dictionary size than TCN-AE. Under the evaluated frozen and zero-shot settings, TSFMs are task-dependent deployment options rather than default replacements for fitted lightweight models.

1 Introduction

The study evaluates whether TSFMs improve industrial monitoring under heterogeneous task definitions, matched protocols, and local resource constraints. It treats deployment value as uncertain when target-normal data and competitive lightweight models are available.

  • Industrial monitoring models must match target labels and evaluation units while meeting local resource constraints.
  • TSFMs promise transferable representations or forecasts with little target-specific adaptation, but their deployment value remains uncertain against lightweight models.
  • The study compares frozen TSFM modes with fitted lightweight models under matched engine- or file-level anomaly-ranking protocols.
  • It tests whether C-MAPSS model ordering changes across degradation-risk horizons and whether zero-shot TSFMs improve BDG2 residual diagnostics.
  • The evaluation also measures latency, memory, and model-size costs accompanying predictive gains under a standardized local workload.
  • The contribution is a protocol-aware empirical deployment comparison, not a new detector, state-of-the-art claim, or universal ranking of model families.

2 Related Work

Related work spans time-series anomaly-detection paradigms, industrial transfer under domain shift, acoustic anomaly detection, building-energy residual monitoring, and time-series foundation models. The paper situates its evaluation within these distinct application settings.

  • Deep time-series anomaly detection includes reconstruction, forecasting, density estimation, contrastive learning, and graph- or attention-based architectures.
  • Industrial transfer methods seek to reduce retraining and labeling requirements, but source–target mismatch must be handled explicitly.
  • MIMII supports anomalous-sound detection under realistic factory noise, while DCASE Task 2 uses normal-only training to detect unknown abnormal sounds.
  • Because BDG2 lacks equipment-fault labels, building-energy monitoring uses forecasting errors and injected perturbations to probe residual sensitivity.
  • MOMENT, Chronos, and TimesFM represent foundation-model approaches for broad time-series tasks, probabilistic forecasting, and zero-shot forecasting, respectively.

3 Methods

The methods use a common scalar monitoring-score interface while preserving domain-specific labels, evaluation units, and interpretation across C-MAPSS, MIMII, and BDG2.

  • The three domains lack a shared native anomaly definition, so results are interpreted within protocol rather than combined in one leaderboard.
  • Each input sequence or window is mapped to a scalar monitoring score while retaining domain-specific interpretation.
  • C-MAPSS uses sliding windows and a degradation-risk proxy, MIMII uses normal-only calibration with file-level evaluation, and BDG2 uses absolute forecasting residuals.
  • The main C-MAPSS analysis uses h = 30 cycles and tests h ∈{15, 20, 30, 40} for sensitivity.
  • Synthetic BDG2 perturbations provide pseudo labels for controlled sensitivity analysis rather than equipment-fault detection.

3.2 Datasets and Preprocessing

The study applies fixed preprocessing and matched data protocols across C-MAPSS, MIMII, and BDG2, with domain-specific labels and evaluation units. Thresholds are calibrated from training scores without using test labels.

  • C-MAPSS: C-MAPSS uses 21 sensor channels, excludes three operating settings, and constructs length-30 windows with stride 1.
  • MIMII: MIMII pump audio is cached as 128 × 32 log-mel tensors, with normalization fitted only on train-normal tensors.
  • BDG2: BDG2 uses a seed-42 panel of 12 electricity meters spanning nine sites and five primary-use categories.
  • Protocol comparison: The protocols use degradation-risk proxy labels for C-MAPSS, native file-level labels for MIMII, and synthetic perturbation diagnostics for BDG2.
  • Scoring and thresholds: Monitoring scores increase with deviation, using negated classical scores, reconstruction error, or absolute residuals depending on model family.
  • Scoring and thresholds: Operating thresholds default to the 95th percentile of training scores, avoiding test-label threshold selection.

3.4 Model Families

The study compares classical one-class estimators, compact neural autoencoders, residual forecasters, and TSFMs in task-specific deployment modes rather than seeking a domain-independent ranking.

  • Classical baselines include Isolation Forest, One-Class SVM, Local Outlier Factor, and PCA reconstruction, using flattened windowed sensor inputs.
  • Compact LSTM and TCN-style autoencoders are trained on scaled normal windows and score deviations by reconstruction error.
  • Energy residual baselines comprise naive, rolling-median, Ridge, and LightGBM forecasters evaluated with forecasting and synthetic pseudo-anomaly diagnostics.
  • MOMENT-small is tested in embedding-plus-Isolation-Forest and reconstruction modes, while Chronos-T5 tiny and TimesFM 2.5 provide zero-shot BDG2 forecasts.
  • The selected TSFMs represent deployment cases across different task modes and domains, excluding fine-tuning and anomaly-specific adapters.

3.5 Cost and Reproducibility Metadata

The study records reproducibility metadata and audits local inference cost using standardized, same-device measurements and protocol-specific aggregation units.

  • Formal runs store protocol and train-selection checksums with Parquet score artifacts so metrics can be recomputed and paired by engine or file.
  • The C-MAPSS cost audit fixes one RTX 4090, float32 inference, shared fold-0 windows, batch sizes 1 and 16, warm-ups, and measured repetitions.
  • The audit reports same-process warm-call latency, peak allocated and reserved VRAM, and serialized state-dictionary size.
  • Dataset archives, configurations, environments, and regenerated paper artifacts are tracked with paths, sizes, checksums, YAML, and locked dependencies.
  • C-MAPSS uses engines, MIMII uses files, and BDG2 uses meters as primary units, with nested injection realizations for BDG2.

3.7 Uncertainty

Uncertainty is estimated within each protocol using aggregation and resampling units matched to engines, files, or meters rather than comparing raw cross-fold scores.

  • C-MAPSS metrics are engine-count-weighted fold means, with paired differences estimated from 10,000 held-out-engine bootstrap resamples within each fold.
  • MIMII reports means and t-based 95% confidence-interval half-widths over five paired file manifests, with stratified paired file-bootstrap diagnostics.
  • BDG2 averages injection seeds within each meter before reporting meter-macro means and dispersion.
  • Chronos-T5 and TimesFM forecast held-out BDG2 targets from a fixed context, while five-seed perturbation experiments keep forecasts and contexts fixed.
  • Synthetic diagnostics report type-specific AUROC, AUPRC, F1, precision, and recall, excluding directly observable missingness from aggregate metrics.

4 Results

Results are protocol-dependent: fitted lightweight models dominate the two anomaly-ranking evaluations, while TimesFM performs best on aligned BDG2 forecasting error and synthetic AUROC, with perturbation difficulty limiting interpretation.

  • Target-Domain Anomaly Ranking: 0.9570/0.8960: TCN-AE reaches fold-weighted AUROC/AUPRC across 100 held-out C-MAPSS engines, versus 0.7310/0.3080 for MOMENT reconstruction.
  • Target-Domain Anomaly Ranking: −0.2260 and −0.5880: MOMENT-minus-TCN-AE paired differences for reconstruction AUROC and AUPRC, with 95% bootstrap intervals excluding zero.
  • Target-Domain Anomaly Ranking: 0.6944±0.0272/0.7277 ± 0.0209: OCSVM has higher five-seed MIMII AUROC/AUPRC means than MOMENT reconstruction at 0.5501 ± 0.0271/0.5890 ± 0.0367.
  • Risk-Horizon Sensitivity: 15, 20, 30, and 40 cycles: C-MAPSS model ordering remains stable across tested degradation-risk horizons, with MOMENT reconstruction behind fitted references at every horizon.
  • Forecasting and Perturbation Sensitivity: MAE 9.3059, RMSE 15.7695, sMAPE 0.1033: TimesFM 2.5 has the lowest meter-macro forecast errors on the aligned 12-meter BDG2 panel.
  • Forecasting and Perturbation Sensitivity: 0.7239: TimesFM 2.5 has the highest aggregate synthetic AUROC point estimate, while synthetic AUPRC remains close across evaluated models.
  • Forecasting and Perturbation Sensitivity: Delayed response, flatline, stuck-at, and seasonal distortion have low AUPRC for all evaluated models, and the aggregate diagnostic cannot establish real building-fault detection.
  • Implementation Cost: 13.368 ms and 691 MB: MOMENT’s batch-1 latency and batch-16 peak allocated VRAM exceed TCN-AE’s 0.294 ms and 2.6 MB on the same RTX 4090.

5 Discussion

The discussion interprets TSFM value as protocol- and task-dependent, while emphasizing scope boundaries in labels, datasets, model coverage, uncertainty, and resource measurement. Synthetic BDG2 gains indicate perturbation sensitivity, not real-fault detection, and local cost comparisons are implementation-specific.

  • When Do TSFMs Pay Off?: BDG2 is the only setting where zero-shot forecasting priors improve selected point estimates, but preferences vary across AUROC, AUPRC, and thresholded metrics.Because BDG2 labels are injected, these results measure synthetic perturbation sensitivity rather than real-fault detection.
  • Threats to Validity: Task heterogeneity prevents a unified benchmark: C-MAPSS uses degradation-risk proxies, MIMII labeled acoustic anomalies, and BDG2 synthetic residual diagnostics.The study compares models only within protocol and does not establish a universal ranking.
  • Threats to Validity: C-MAPSS high-risk labels derive from each engine’s observed terminal cycle rather than native anomaly annotations, while stride-1 windows remain highly dependent within engines.Engine-disjoint folds prevent engine and exact-window leakage; clustered bootstrap respects the remaining grouping.
  • Threats to Validity: Synthetic BDG2 diagnostics cover ten event types on 12 selected meters and five injection seeds, but do not measure real equipment faults or represent all buildings.The panel broadens coverage beyond one meter but is not a random sample of all buildings.
  • Threats to Validity: TSFM conclusions are conditional on MOMENT-small, Chronos-T5 tiny, and TimesFM 2.5 200M under frozen or zero-shot settings; larger or adapted variants could differ.The evaluation does not assess TSFM fine-tuning, anomaly-specific adapters, or alternative context construction.
  • Threats to Validity: BDG2 reports meter dispersion rather than population-level confidence intervals, with injection seeds nested within the fixed panel.Deep models are also not repeatedly initialized within every C-MAPSS fold.

6 Conclusion

Across the evaluated protocols, fitted lightweight models outperform frozen MOMENT-small modes for anomaly ranking, while forecasting TSFMs show limited, metric-dependent BDG2 advantages. The authors therefore frame foundation models as task-dependent deployment choices, with stronger claims requiring real event labels and adapted evaluation.

  • Conclusion: Fitted lightweight models remain stronger than frozen MOMENT-small modes across engine-disjoint C-MAPSS and paired MIMII protocols, including four C-MAPSS risk horizons.Forecasting TSFMs show limited BDG2 advantages in forecast error and synthetic AUROC, but not AUPRC or thresholded F1.
  • Conclusion: TSFMs should be evaluated as task-dependent deployment choices rather than default replacements for fitted models.Compact fitted models remain strong references when target-normal data and resource constraints matter, while zero-shot forecasters merit evaluation when residual forecasting is valuable.
  • Conclusion: Synthetic diagnostics can screen forecasters but cannot certify detectors without real event labels and end-to-end resource accounting.Deployment decisions should retain target-normal baselines and report calibration boundaries alongside ranking metrics.
Loading 2608.22968v1…