PaperlayerView exact revision on arXiv

Source-linked AI summary

Time-Series Foundation Model Embeddings for Remaining Useful Life Estimation

Amir El-Ghoussani, Michele De Vita, Ronald Naumann, Vasileios Belagiannis

arXiv:2606.11990v3cs.LGcs.AI

TL;DR

Accurate remaining useful life prediction from multivariate sensor streams is important for predictive maintenance, but existing approaches often require extensive feature engineering or large labeled datasets. This paper uses frozen Chronos-2 embeddings with a lightweight regression head and finds the approach outperforms evaluated baselines across two industrial device datasets, with longer histories improving performance.

  • Problem

    Accurate RUL prediction from multivariate sensor streams is important for effective predictive maintenance, while many learning-based approaches require extensive feature engineering or large labeled datasets.

  • Method

    The approach extracts context-dependent embeddings from multivariate sensor windows using frozen Chronos-2 and trains a lightweight regression head for RUL prediction.

  • Results

    Chronos-2 adaptation achieves the best overall performance among evaluated models across both devices, while performance improves significantly with longer context and saturates beyond L = 80.

  • Takeaways & Limitations

    A frozen pretrained time-series foundation model can serve as an effective feature extractor for industrial RUL estimation with a simple lightweight regressor.

  • Takeaways & Limitations

    The study leaves robustness under missing sensors, operating-regime shifts, and low-label settings for future evaluation.

Abstract

from arXiv · show

Remaining Useful Life (RUL) prediction is essential for industrial predictive maintenance, yet many learning-based approaches rely on extensive feature engineering or large labeled datasets to train task-specific sequence models. In this work, we introduce a lightweight learning approach, in which we leverage a frozen pretrained time-series foundation model (TSFM) and combine it with a small regression head for RUL estimation from multivariate sensor streams. More specifically, we use Chronos-2 as a frozen backbone to extract context window features and train a lightweight regression neural network for RUL prediction. Experiments on real-world industrial sensor data from two device types show that Chronos-2 features consistently improve over recurrent, convolutional, Transformer-based, and gradient-boosting baselines under the same preprocessing and evaluation protocol. We further analyze the impact of context length and find that performance improves significantly with longer histories, indicating that TSFM representation offer a practical and data-efficient alternative for RUL estimation in industrial settings.

I. INTRODUCTION

Accurate RUL prediction from multivariate sensor streams is central to predictive maintenance, but conventional approaches depend on handcrafted features or task-specific sequence models. This work instead extends pretrained TSFM representations to supervised RUL regression by adapting CHRONOS-2 with a lightweight regression head.

  • Motivation: RUL estimates forecast time until asset failure from current condition, supporting advance intervention and reducing unexpected downtime and associated costs.Accurate RUL prediction from multivariate sensor streams is identified as a major focus in prognostics.
  • Existing approaches: Traditional data-driven RUL estimation relies on handcrafted features or end-to-end sequence models trained on multivariate historical sensor measurements.Related approaches include LSTM and GRU models for capturing temporal patterns.
  • Existing approaches: Pretraining on unlabeled sensor data through contrastive and self-supervised representation learning can relax labeling requirements before supervised RUL regression.Unsupervised anomaly detection provides a related label-efficient temporal monitoring paradigm.
  • Time-series foundation models: TSFMs such as Chronos and CHRONOS-2 are probabilistic transformer-based forecasters, while Kairos learns multivariate time-series representations through masked token modeling.CHRONOS-2 uses a decoder-only transformer that tokenizes input time series into discrete bins and supports in-context learning without fine-tuning.
  • Proposed approach: The proposed method treats RUL prediction as supervised regression by loading past sensor measurements into CHRONOS-2, extracting backbone representations, and training a lightweight regression head.This approach extends pretrained TSFM representations beyond their traditional forecasting use case while keeping adaptation simple.

II. METHOD

The method processes the complete multivariate sensor-history sequence to estimate RUL at each time step. It maps the input sequence to RUL estimates using a frozen pretrained backbone and a trainable regression head.

  • Input representation: The method uses the complete history of multivariate sensor measurements as input.The sequence is represented as X ∈ R^T × D.
  • Input representation: Each time step contains a D-dimensional sensor-reading vector, and T denotes the total sequence length.The input sequence is X = {x_t}, with x_t representing measurements at time t.
  • Model architecture: RUL estimates are produced at every time step by combining a frozen pretrained backbone Φ(·) with a trainable regression head g_ϕ(·).The two-model mapping converts the input sequence into a sequence of RUL estimates y_t.

A. Data Preprocessing and Label Generation

The preprocessing pipeline converts irregular multivariate sensor measurements into regularly sampled, cleaned, and normalized sequences suitable for pretrained TSFMs. RUL labels are derived from maintenance-log failure times, capped at 1000 days, and unavailable post-repair timestamps are excluded.

  • Resampling: Irregular sensor records are linearly interpolated onto a uniform time grid with step size ∆t, producing a multivariate sequence with regular intervals.This accommodates pretrained TSFMs, including CHRONOS, which require regularly sampled data.
  • Resampling: Intervals with gaps exceeding ∆tmax are discarded, and interpolation is restricted to valid ranges to prevent artifacts from long outages.The exclusion condition is τk+1 −τk > ∆tmax.
  • Cleaning and normalization: Post-resampling preprocessing removes NaN measurements, clips values to the 1st and 99th percentiles, and standardizes channels to zero mean and unit variance.All preprocessing statistics are computed exclusively on the training split to prevent data leakage.
  • RUL Label Construction: RUL labels use maintenance logs to represent the time remaining until the next failure event.The next failure is determined from the set of failure timestamps Trep.
  • RUL Label Construction: RUL values are expressed in days and capped at ymax = 1000 days, while timestamps without subsequent repair events are excluded from supervised training.The cap is applied as yt ←min(yt, ymax).

B. Model Architecture

The model uses frozen CHRONOS-2 as a decoder-only Transformer feature extractor for windowed multivariate sensor sequences, then applies a lightweight trainable MLP head to estimate RUL. Context-window embeddings capture temporal dependencies, while the head uses ReLU activations and nonnegative output enforcement.

  • Context Extraction: CHRONOS-2 serves as a fixed feature extractor using its pretrained decoder-only Transformer architecture.The chronos2 checkpoint is used with all TSFM parameters frozen.
  • Context Extraction: The frozen TSFM processes a sensor sequence window of L timesteps to produce hidden representations that capture temporal context and dependencies.The representations are organized as H = [h_t−L, …, h_t], with hidden dimension h.
  • RUL Estimation: RUL at time t is estimated from the corresponding context-window embedding h_t−L:t using a trainable regressor head g_ϕ.Only the head parameters ϕ are optimized.
  • RUL Estimation: The regressor is a two-layer MLP with hidden width m, ReLU activation, dropout rate p, and a final ReLU enforcing ŷ_t ≥ 0.Dropout is applied after the first hidden layer.

III. EXPERIMENTS · A. Dataset and experimental setup

Experiments use multivariate sensor streams from two industrial devices, with RUL labels derived from repair logs after standardized preprocessing. Evaluation uses a chronological 85:15 split that excludes boundary-overlapping windows and reports MAE and MSE on held-out data.

  • A. Dataset and experimental setup: The experiments use industrial sensor datasets provided by Nokia Solutions and Networks GmbH & Co. KG, Germany.
  • A. Dataset and experimental setup: The datasets contain multivariate sensor streams from two devices, designated Device A and Device B.
  • A. Dataset and experimental setup: Sensor streams are resampled to a uniform time grid during preprocessing.
  • A. Dataset and experimental setup: Each sensor channel is normalized before model evaluation.
  • A. Dataset and experimental setup: RUL labels are computed from repair logs as described in Sec. II-A.
  • A. Dataset and experimental setup: A chronological split assigns training windows to t ≤Ttrain and test windows to t > Ttest, targeting an 85:15 ratio.
  • A. Dataset and experimental setup: Windows overlapping the split boundary are discarded to prevent temporal leakage.
  • A. Dataset and experimental setup: Held-out test performance is reported using Mean Absolute Error (MAE) and Mean Squared Error (MSE).

B. Implementation · C. Baselines

The approach is trained as compact device-specific models with MSE and Adam, while comparisons span non-sequential regressors and sequential neural, boosting, convolutional, and attention-based baselines under aligned preprocessing and evaluation protocols.

  • B. Implementation: Training uses MSE loss, Adam with a 10−3 learning rate, up to 50 epochs, and batches of 64.Fine-tuning takes about 2 hours on one NVIDIA A6000 GPU.
  • B. Implementation: Separate models contain approximately 300K parameters for device A and 250K parameters for device B.The models are trained independently for the two device types.
  • C. Baselines: Baselines include linear regression and random forests trained on normalized sensor vectors with the same capped targets and train/test splits.These methods represent the non-sequential comparison setting.
  • C. Baselines: LSTM and GRU regressors process length-L context windows and predict RUL from the final recurrent state through a linear readout.They are trained as lightweight sequential neural baselines.
  • C. Baselines: Gradient boosting uses window features comprising per-channel statistics, the last value, first differences, and a linear trend slope.The feature map includes mean, std, min, max, and quantiles, and uses the same L, preprocessing, and chronological splits.
  • C. Baselines: A Temporal Convolutional Network and Transformer encoder map context windows to scalar RUL predictions using temporal pooling followed by a linear output layer.Both are evaluated as convolutional or attention-based sequence models.
  • C. Baselines: All baseline hyperparameters are tuned on an approximately 10% validation split and evaluated with the same metrics as the proposed method.The comparison therefore uses a shared evaluation protocol across baseline families.

D. Performance on 5-step windows · E. Effect of Context Length

On 5-step windows, neural sequence models substantially outperform non-neural approaches, while the CHRONOS-2 adaptation further improves performance on both devices. Increasing context length benefits CHRONOS-2 most strongly through L = 80, after which performance saturates and remains above other baselines.

  • D. Performance on 5-step windows: Neural sequence models substantially outperform non-neural approaches on 5-step windows for both Device A and Device B.This indicates that even short context windows carry significant information.
  • D. Performance on 5-step windows: The CHRONOS-2 adaptation further improves performance across both devices by extracting temporal-context embeddings and adding a learned regression component.The supplied passage describes the adaptation mechanism but truncates before reporting exact Table I values.
  • E. Effect of Context Length: Context-length experiments vary L with fixed resampling, chronological splits, frozen CHRONOS-2 weights, and retrained MLP heads.Each CHRONOS-2 setting is trained for 50 epochs, while baselines are retrained and re-tuned for each L.
  • E. Effect of Context Length: CHRONOS-2 performance improves significantly as context length increases, with the most substantial gains observed up to L = 80.Beyond L = 80, performance saturates.
  • E. Effect of Context Length: Other baselines show only mild fluctuations as context length increases, indicating limited use of longer context windows.The figure likewise summarizes mild baseline fluctuations across context lengths.
  • E. Effect of Context Length: TCN and Transformer are the strongest non-TSFM baselines but still significantly underperform the proposed method across all context lengths.Figure 2 also identifies TCN and Transformer as the strongest baselines while showing significant improvement for Ours.

F. Ablation: Regression head

The ablation shows that frozen CHRONOS-2 embeddings provide most of the performance gain, while a 2-layer MLP adds useful nonlinearity and deeper capacity offers no significant further improvement.

  • Regression head capacity: A linear head on frozen CHRONOS-2 embeddings achieves MAE 60 versus 88 for the TCN baseline.This indicates that the pretrained backbone provides a strong representation for RUL estimation.
  • Regression head capacity: The 2-layer MLP outperforms the linear head, showing that some nonlinearity benefits mapping CHRONOS-2 embeddings to RUL predictions.
  • Regression head capacity: Increasing the regression head to 4 layers yields no significant further improvement over the 2-layer MLP.The ablations keep the CHRONOS-2 backbone frozen and use identical preprocessing, context length, and training budget on device A.

IV. CONCLUSION

The paper shows that frozen CHRONOS-2 embeddings with a lightweight regressor can support RUL estimation from multivariate sensor windows. It also identifies calibrated uncertainty, improved sensor handling, and robustness evaluation as future priorities.

  • Conclusion: Frozen CHRONOS-2 extracts context-dependent embeddings from multivariate sensor windows for lightweight RUL regression.The approach uses the pretrained backbone as a feature extractor rather than training a task-specific sequence model end to end.
  • Conclusion: On the Nokia dataset, the approach achieved lower error than a broad set of classical and deep learning baselines.
  • Future Work: Future work will use CHRONOS2 probabilistic outputs to provide calibrated uncertainty for maintenance decisions.
  • Future Work: Future work will improve multivariate tokenization and channel handling for heterogeneous sensors.
  • Future Work: Future evaluations will test robustness under missing sensors, operating-regime shifts, and low-label settings.
Loading 2606.11990v3…