PaperlayerView exact revision on arXiv

Source-linked AI summary

Long Horizon Transformer Quantile Fault Prediction for Multi Site Industrial Predictive Maintenance

David J Poland, Daniele Ravi, Na Helian

arXiv:2609.04840v1cs.AI

TL;DR

Long-horizon predictive maintenance needs models that distinguish gradual degradation from normal regime variation over day-scale planning windows. The paper proposes TQRNN30d, combining dual-stage conditional-quantile features with multi-stream temporal fusion for supervised fault-occurrence forecasting. It leads the evaluated baselines across 7-, 14-, and 30-day horizons, while evidence remains limited to held-out machines within a homogeneous nine-facility fleet.

Abstract

from arXiv · show

Long-horizon predictive maintenance requires models to distinguish slowly evolving degradation from normal operating-regime variation over planning windows measured in days rather than hours. This paper evaluates whether an explicit conditional-quantile representation provides an informative classifier interface for this problem. The proposed TQRNN30d framework combines a dual-stage quantile regression neural network (QRNN) feature extractor with a multi-stream temporal fusion classifier. Each hourly word of 81-channel machine behaviour is mapped to a 324-dimensional quantile-state representation, and 720 ordered hourly words form the 30-day document supplied to the long-horizon model. The classifier fuses quantile states with dynamic covariates, channel-level static metadata, and a 168-hour latent-history stream using gated residual processing, causal recurrent encoding, and metadata-conditioned cross-modal attention. A bounded instability-aware signal derived from sustained one-word-ahead prediction-error divergence provides auxiliary memory modulation at the longest horizon. Evaluation uses a machine-disjoint 43/14/15 train/validation/test allocation across 72 machines in nine manufacturing facilities. At 30 days, TQRNN30d achieves 79.97% F1, 80.18% recall, 81.82% precision, 82.39% accuracy, and 0.820 ROC-AUC. It leads all 18 evaluated baselines at the 7-, 14-, and 30-day fixed-threshold comparisons, with the largest F1 advantage at 14 days. The results support held-out-machine performance within the observed homogeneous nine-facility fleet, but do not establish unseen-site, cross-equipment, or cross-sector generalisation.

I. INTRODUCTION

Long-horizon industrial predictive maintenance addresses gradual distributional degradation, heterogeneous sensing, regime transitions, and evaluation leakage that complicate weeks-ahead forecasting. TQRNN30d treats the input representation as central, using conditional quantiles for supervised 7-, 14-, and 30-day fault-occurrence forecasting.

Why long horizons, and why multiple sites

  • 30-day fleet planning motivates forecasting fault occurrence weeks ahead rather than issuing only hour-ahead alarms.The paper links long horizons to maintenance scheduling across geographically separated plants.Source: Why long horizons, and why multiple… · p. 1
  • Long horizons intensify gradual distributional change, causal multi-rate sensor fusion, non-stationary regimes, and the limits of ordinary sequential classifiers.The cited challenges include variance growth, tail movement, envelope widening, incompatible sampling rates, regime transitions, and missing distributional information.Source: Why long horizons, and why multiple… · p. 1

Scope of the task

  • TQRNN30d forecasts whether at least one confirmed fault occurs within 7-, 14-, or 30-day future horizons from a completed observation document.The task is supervised Normal/Abnormal forecasting rather than direct RUL or degradation-index regression.Source: Scope of the task · p. 2

Contributions

  • The framework combines an 81-channel quantile representation with a multi-stream Temporal Fusion Transformer and an auxiliary instability-aware memory pathway.The contributions distinguish the conditional-quantile interface, dual-stage representation, heuristic instability modulation, and machine-disjoint evaluation.Source: Scope of the task · p. 2

Related work and gap

  • The literature gap is a long-horizon method centered on input representation and validated with partitioning strict enough to withstand leakage concerns.Prior work often uses short horizons, asset-specific settings, or benchmark datasets, while reported results are difficult to compare across protocols.Source: A. Data-Driven Predictive Maintenance · pp. 2–3

A. Why a Quantile Encoding Rather Than Point Summaries

  • Quantile regression preserves distributional widening, asymmetric tails, and early degradation envelopes rather than reducing sensor behavior to point forecasts.Here quantile regression serves as a classifier interface, not as the final forecasting target.Source: B. Quantile Regression for Prognostics · p. 3

D. Temporal Fusion and Attention Models

The paper distinguishes present-state anomaly detection from supervised prospective fault forecasting. Its temporal-fusion design uses sequential and contextual information while treating sustained error divergence as a heuristic instability cue.

Temporal Fusion Transformer

  • TFT contributes gated residual processing, static covariate conditioning, temporal variable selection, and attention for multi-horizon sequence modelling.These properties are presented as suitable for machine evolution together with contextual operating factors.Source: D. Temporal Fusion and Attention Models · p. 3

Task distinction

  • Fault forecasting asks whether a labelled abnormal event will occur within a future window, unlike anomaly detection, which evaluates present deviation.The forecasting task is supervised and evaluated on lead time as well as separability.Source: E. Anomaly Detection Versus Fault… · pp. 3–4

Prospective prediction

  • TQRNN30d uses evolving distributional structure to predict future abnormal behaviour rather than merely detecting current deviation.The framework is prospective anomaly prediction within a future forecasting horizon.Source: E. Anomaly Detection Versus Fault… · p. 4

Comparative baselines

  • Autoencoder and GAN anomaly-oriented models are adapted to the same future fault-occurrence target for like-for-like baseline comparison.The comparison tests whether conventional anomaly-oriented representations remain competitive after shifting from present detection to long-horizon prediction.Source: E. Anomaly Detection Versus Fault… · p. 4

Instability Indicators

  • The instability-aware gate is explicitly heuristic and does not estimate a formal dynamical invariant such as a Lyapunov exponent.Classical diagnostics require reconstruction and stationary records unavailable in the described causal deployment.Source: F. Instability Indicators · p. 4

Deployment context

  • The deployment contains 72 machines across nine facilities within one organisation and a homogeneous production-equipment fleet.The sensor and machine context is multi-facility but not presented as a cross-organisation or cross-equipment setting.Source: A. Deployment Context and Asset… · p. 4

2) Operational sensor-ablation context:

The operational sensor-ablation evidence indicates redundancy and resilience to isolated sensor loss, while the temporal interface converts multi-rate signals into ordered hourly quantile-state documents.

Operational sensor-ablation context:

  • Approximately 94% or higher detection rates were retained for nonthermal channels under single-sensor removal.The thermal group was less central to the mechanically coupled fault-discriminative pathway.Source: 2) Operational sensor-ablation context · p. 4
  • Cross-group fusion is supported as resilient to isolated sensor loss, but the ablation is not an unseen-site generalisation test.It is distinct from the 15-machine held-out cohort used for formal classifier comparison.Source: 2) Operational sensor-ablation context · p. 4

Temporal alignment

  • Causal zero-order hold projects asynchronous streams onto a fixed 20 ms reference grid without introducing future measurements.Channels retain each observed value until the next native sample becomes available.Source: 1) Multi-rate causal alignment · p. 5

Word and document representation

Latent history

  • A 168-hour latent-history stream gives the classifier recent operating memory without reprocessing the complete raw trace.Documents refresh at one-hour intervals while temporal ordering is preserved.Source: 1) Multi-rate causal alignment · p. 5

Quantile feature extraction

  • The dual-stage extractor refines four retained quantiles per channel, yielding the 324-dimensional word-level representation.The second stage adds interior quantiles around the median while retaining the interquartile envelope.Source: 1) Multi-rate causal alignment · p. 5

D. Label Construction

The study defines future fault labels from confirmed operational events within strictly future horizons while enforcing causal, machine-disjoint evaluation. Ground truth uses multiple operational sources, with explicit exclusions and scope boundaries for the long-horizon deployment.

B. Dual-Stage Quantile Feature Extraction

The dual-stage extractor converts each hourly 81-channel word into a distribution-aware quantile state, which the classifier combines with contextual and historical streams. The design targets distributional changes while preserving causal long-horizon sequence processing.

Dual-stage extraction

Representation

Multi-stream fusion

Prediction head

D. Instability-Aware Memory Gate

The instability-aware memory gate derives a bounded auxiliary signal from sustained one-word-ahead prediction-error divergence and uses it to modulate causal temporal memory. It supplements, rather than replaces, the retained QRNN representation and final classifier.

Signal construction

  • The pathway uses retained channel-level quantile states to predict each auxiliary channel summary one hourly word ahead.Predictions use only information strictly before word k.Source: D. Instability-Aware Memory Gate · p. 8

Persistence criterion

  • The gate identifies sustained local divergence by requiring three consecutive threshold exceedances before activation.The evaluated threshold is λ0 = 0.2.Source: D. Instability-Aware Memory Gate · p. 9

Signal aggregation

  • The aggregate divergence signal is restricted to degradation-sensitive channels, weighted by learned non-negative channel weights, and bounded before temporal gating.These steps limit the influence of contextual or instrumentation-related variation and constrain the modulation magnitude.Source: D. Instability-Aware Memory Gate · p. 9

Memory modulation

  • The bounded signal modulates the causal GRU update gate so persistent divergence biases memory toward retaining more of the previous hidden state.The classifier then combines the quantile, dynamic-covariate, and historical-latent streams through metadata-conditioned fusion.Source: D. Instability-Aware Memory Gate · p. 9, Figure 3 · p. 9

Role in the classifier

  • The instability pathway is auxiliary: the retained QRNN representation remains the primary predictive basis, and its independent contribution was not isolated by a dedicated ablation.The final head emits structured hourly forecast-interval probabilities before the horizon-level decision.Source: D. Instability-Aware Memory Gate · p. 9, Figure 3 · p. 9

1) Interpretation and limitations of instability-aware gating:

Instability-aware gating detects sustained divergence in prediction-error trajectories but does not identify the physical cause. Its influence is constrained by channel selection, persistence, bounded modulation, and development-data calibration, yet sustained regime changes may still activate it.

Interpretation

  • The gate may respond to progressive degradation, sensor drift, material transitions, or sustained operating-regime changes.It should therefore be interpreted as temporal modulation rather than an independent fault detector.Source: 1) Interpretation and limitations of… · p. 9

Mitigations

  • Restricting aggregation to degradation-sensitive channels and bounding the modulation reduces sensitivity to non-degradation disturbances.The persistence criterion also suppresses isolated transient excursions.Source: 1) Interpretation and limitations of… · p. 9

Residual limitation

  • Sustained non-fault regime changes may still activate the pathway despite the persistence requirement.This is especially relevant during material, lubrication, or operating-condition transitions without associated mechanical failure.Source: 1) Interpretation and limitations of… · p. 9

Configuration

  • The implemented gate uses λ0 = 0.2, three consecutive hourly exceedances, ϵ = 10^-3, and retained quantile context α = 0.25, 0.40, 0.60, 0.75.These are implementation parameters rather than physical constants.Source: 2) Instability-gate operating… · p. 9

Transferability

  • Threshold and persistence settings were selected on development data and are not claimed to transfer without recalibration to another machine population.The held-out test partition was excluded from parameter selection.Source: 2) Instability-gate operating… · p. 9

E. Training Objective

The training objective first constructs the upstream quantile representation, then trains the long-horizon classifier as a document-level multi-output binary predictor. Evaluation compares this framework with 18 complementary model families under common task and data foundations.

Training objective

  • The two QRNN stages use pinball quantile loss, after which the TFT classifier is trained on the QRNN representation.The classifier predicts failure probabilities for each forecast interval.Source: E. Training Objective · p. 9

Classification loss

  • The classifier uses interval-wise binary crossentropy with clipped predicted probabilities for numerical stability.The target and predicted failure probability are defined for each training document and forecast interval.Source: E. Training Objective · pp. 9–10

Class imbalance

  • Class-weighted BCE addresses sparse confirmed-failure intervals, with inverse-prevalence weights computed from training data only.The weights remain fixed during validation and testing.Source: E. Training Objective · p. 10

Baseline comparison

  • The comparison spans 18 baselines across kernel, instance-based, tree, recurrent, feed-forward, reconstruction, attention, and pretrained-transformer families.These families test temporal modelling, anomaly-detection, long-range attention, and pretrained sequence-classification alternatives.Source: 1) Baseline selection and coverage · p. 10
  • Attention baselines include Transformer, Transformer-XL, and TFTfull, while BERT, RoBERTa, DistilBERT, and ALBERT test pretrained sequence-model alternatives.TFTfull is identified as the direct architectural ancestor and most informative single comparator.Source: 1) Baseline selection and coverage · p. 10

2) Adaptation of BERT-family sequence baselines:

The BERT-family rows are retained as architecture-family comparators under the same long-horizon task and held-out population, without unsupported claims about their adapter details. The broader evaluation fixes data-processing and threshold choices through training or validation data, while several transfer and repeated-seed analyses remain unreported.

BERT-family baselines

  • BERT, RoBERTa, DistilBERT, and ALBERT are retained sequence-model baselines rather than newly specified token-level implementations.The experimental record does not provide sufficient adapter detail to justify new tokenisation or positional-embedding claims.Source: 2) Adaptation of BERT-family sequence… · p. 10

Comparative fairness

  • All models use machine-disjoint training, validation, and held-out test assignments with preprocessing estimated from applicable development data only.This enforces a common data and evaluation foundation.Source: 3) Controlled comparison protocol · p. 10
  • Hyperparameters, stopping criteria, imbalance treatment, and decision thresholds are selected using training or validation data before held-out evaluation.The study does not claim identical search spaces or equal numbers of trials across model families.Source: 3) Controlled comparison protocol · p. 10

Scope of evidence

  • The completed programme excludes rolling-origin evaluation, leave-one-site-out transfer, additional long-sequence baselines, matched one-component ablations, and systematic multi-seed retraining.Headline results therefore come from one completed training and held-out evaluation run per configuration.Source: 4) Scope of the reported experiments · p. 11, C. Implementation · p. 11

TFT input regimes

  • TFTfull receives the complete MLP–QRNN quantile-state representation, whereas TFTflat removes that hierarchy and uses raw flat input for the ablation.They are the same network under different input regimes.Source: 5) Naming of TFT configurations · p. 11

Evaluation metrics

  • Reported performance uses thresholded F1-score, recall, precision, and accuracy, alongside ROC–AUC and a multi-site precision–recall profile area.The thresholded metrics are unweighted means across the corresponding held-out cohort.Source: B. Metrics · p. 11

D. Modelling Assumptions

The study states assumptions about labels, sensors, and deployment stability, while reporting performance under fixed evaluation and decision-threshold procedures. Across evaluated horizons, TQRNN30d outperforms RoBERTa, with the largest F1 difference at 14 days.

B. Component Ablation

The ablations separate day-scale pipeline evidence from TFT-side progression under flat-input conditions, while discrimination and computational comparisons assess the full model more broadly. The evidence supports gains from the integrated representation and fusion pathway without establishing single-component causal attribution.

E. Operational Deployment Context

Deployment occurred across nine facilities within a broader maintenance programme, but the operational changes cannot be attributed independently to the predictive framework. The evaluation supports held-out-machine performance within a homogeneous fleet, not broader transfer or causal deployment claims.

Related papers

Loading 2609.04840v1…