PaperlayerView exact revision on arXiv

Source-linked AI summary

Forecasting remaining useful life: Interpretable deep learning approach via variational Bayesian inferences

Mathias Kraus, Stefan Feuerriegel

arXiv:1907.05146v2cs.LGstat.ML

TL;DR

Black-box machine-learning models can limit insight into the rules behind RUL forecasts, despite their use of sensor data. The paper proposes a structured-effect neural network and reports lower forecast error than distribution-based approaches while retaining interpretability and accountability.

  • Problem

    Machine-learning models often operate as black boxes, impeding insight into the decision rules behind RUL forecasts.

    Source: §1 · p. 2

  • Method

    A structured-effect neural network combines deep-learning flexibility with probabilistic interpretability, with parameters estimated through variational Bayesian inference.

    Source: §1 · p. 3

  • Results

    51.60% lower forecast error than distribution-based approaches; deep learning slightly surpasses the model, which retains interpretability and the accountability of simpler probabilistic approaches.

    Source: §1 · p. 4

  • Takeaways & Limitations

    The approach offers interpretable, accountable RUL forecasts for decision support.

    Source: §1 · p. 4

    The paper notes that Bayesian approximation via dropout does not provide uncertainty estimates for model coefficients.

    Source: §5.4 · p. 27

Abstract

from arXiv · show

Predicting the remaining useful life of machinery, infrastructure, or other equipment can facilitate preemptive maintenance decisions, whereby a failure is prevented through timely repair or replacement. This allows for a better decision support by considering the anticipated time-to-failure and thus promises to reduce costs. Here a common baseline may be derived by fitting a probability density function to past lifetimes and then utilizing the (conditional) expected remaining useful life as a prognostic. This approach finds widespread use in practice because of its high explanatory power. A more accurate alternative is promised by machine learning, where forecasts incorporate deterioration processes and environmental variables through sensor data. However, machine learning largely functions as a black-box method and its forecasts thus forfeit most of the desired interpretability. As our primary contribution, we propose a structured-effect neural network for predicting the remaining useful life which combines the favorable properties of both approaches: its key innovation is that it offers both a high accountability and the flexibility of deep learning. The parameters are estimated via variational Bayesian inferences. The different approaches are compared based on the actual time-to-failure for aircraft engines. This demonstrates the performance and superior interpretability of our method, while we finally discuss implications for decision support.

1. Introduction

The paper targets interpretable RUL forecasting for preemptive maintenance, proposing a structured-effect neural network that combines deep-learning flexibility with probabilistic accountability. On aircraft-engine data, it reduces forecast error relative to distribution-based approaches, although deep learning slightly surpasses it.

1. Introduction

  • Preemptive maintenance uses predicted time-to-failure to guide advance repairs or replacement, aiming to reduce downtime and costs.Source: §1 · p. 2
  • Sensor-based machine learning captures nonlinear relationships for RUL forecasts, but black-box decision logic can impede insight into its underlying rules.Interpretability is defined here as transparency of the model’s own decision logic.Source: §1 · pp. 2–3
  • The structured-effect neural network combines deep-learning flexibility with distributional interpretability, and its parameters are estimated via variational Bayesian inference minimizing Kullback-Leibler divergence.Source: §1 · p. 3
  • The evaluation uses the public Turbofan Engine Degradation Simulation dataset, with sensor measurements from aircraft engines.Source: §1 · p. 3
  • 51.60% lower forecast error than distribution-based approaches; deep learning slightly surpasses the model, which remains interpretable with probabilistic approaches’ accountability.The comparison uses aircraft-engine time-to-failure data.Source: §1 · p. 4

2. Background

The background frames remaining useful life estimation as support for preemptive maintenance and situates the work amid probabilistic and machine-learning approaches. It distinguishes post-hoc explanations from models that encode relationships in advance, motivating structured predictions for traceability and comparison with prior knowledge.

2.1. Probabilistic lifetime models

  • Probabilistic lifetime models offer interpretable, reliable estimates but often rely on event data and may ignore sensor data’s prognostic capacity.They can be extended with covariates, but then remain constrained by their mathematical structure; proportional hazards models combine a baseline lifetime function with covariate effects.Source: §2.1 · p. 6

2.2. Machine learning in lifetime predictions

  • The paper proposes combining structural predictions and deep learning to balance interpretability with prognostic capacity.The background notes that deep neural networks can reduce forecast error but are often black boxes with limited interpretability.Source: §2.2 · p. 7

2.3. Verification in machine learning

  • Formal verification can establish specification consistency, but verification approaches face scalability limits, especially for modern deep-learning systems [e. g. 40].The background also describes worst-case searches during evaluation as one approach to finding cases where model behavior may be undesirable [e. g. 39].Source: §2.3 · p. 8

2.4. Explainable vs. interpretable machine learning

  • Explainable machine learning provides post-hoc accounts of predictions without elucidating the mechanisms by which the model works.Examples include local linear approximations such as partial dependence plots [41] and decomposing predictions into feature contributions with SHAP values [42].Source: §2.4 · p. 8
  • Explainable methods can visualize what a model has learned, but reveal little about its estimation process and are limited to local input or prediction neighborhoods.The passages describe visualizations as a qualitative means of examining learned behavior.Source: §2.4 · p. 8
  • Interpretable machine learning encodes structure a priori, making model mechanisms and global feature relationships explicit across possible inputs.Relationships among individual features or outcomes for average cases are explicitly modeled.Source: §2.4 · p. 8
  • Linear models are prevalent in safety-critical applications requiring complete estimation traceability, allowing estimates to be compared with prior knowledge or used to gain insights.The passages distinguish traceability of estimation from explanations of predictions.Source: §2.4 · p. 8

3. Methods

The methods aim to combine deep learning’s out-of-sample forecasting performance with the interpretability of probabilistic failure models. They also set out to interpret how sensor types and usage profiles relate to forecasts and compare the approach with several alternative methods.

3. Methods

  • The proposed approach seeks favorable out-of-sample performance and interpretability while interpreting the relevance of sensor types and usage profiles to forecasts.It combines the interpretable feature of probabilistic failure-rate models with deep learning.Source: §3 · pp. 8–9
  • Forecasting performance is compared with naïve empirical estimates, probabilistic approaches, traditional machine learning, feature-engineered time-series methods, and deep neural networks.Source: §3 · p. 9

3.1. Na¨ıve empirical estimation of remaining useful life

  • Naïve empirical RUL estimation starts from past machinery lifetimes and estimates total lifetime using their sample mean.The estimated RUL subtracts the time in use since the last maintenance from that estimated lifetime.Source: §3.1 · p. 9

3.2. Probabilistic lifetime models

Probabilistic lifetime models estimate remaining useful life from historical lifetimes, conditioning the expected lifetime on the machine having already operated for t time steps. The section considers Weibull and log-normal lifetime distributions.

3.2. Probabilistic lifetime models

  • The section uses Weibull and log-normal probability distributions to model machinery lifetimes from historical times between failures.Their distribution parameters are estimated from past event data.Source: §3.2 · pp. 9–10
  • RUL at time t is the expected total lifetime conditional on survival beyond t, minus the time already elapsed.This conditions the estimate on the time since the last maintenance event.Source: §3.2 · p. 10
  • For the Weibull conditional expectation, the authors report no known closed-form solution and approximate it using Markov chain Monte Carlo.They use sampling to approximate the expected remaining lifespan for both Weibull and log-normal models.Source: §3.2 · p. 10

3.3. Traditional machine learning

Traditional machine-learning models predict RUL by minimizing the difference between forecasts and true RUL. They can use current sensor readings or engineered features that summarize sensor history.

3.3. Traditional machine learning

  • The models estimate RUL from inputs and choose parameters by minimizing the deviation between predicted and true RUL.The section includes linear baselines such as ridge regression, lasso, and elastic net, alongside non-linear models such as random forest and SVR.Source: §3.3 · pp. 10–11
  • Models can use current sensor measurements alone, but that choice ignores the trajectory of historic sensor data.Feature engineering offers an alternative by condensing the past time series into an input vector.Source: §3.3 · p. 11
  • Aggregation functions summarize past sensor measurements into a fixed number of features for predictive models.The authors adapt aggregation choices from Mosallam et al., including statistics intended to capture different patterns such as peaks, variation, or sustained levels.Source: §3.3 · p. 11
  • Traditional-model hyperparameters are selected with grouped 10-fold cross-validation that keeps maintenance cycles intact across training and validation.The group approach also prevents the same maintenance cycle from appearing in both sets.Source: §3.3 · p. 12

3.4. Recurrent neural network

Recurrent neural networks process sensor sequences by updating a hidden state that summarizes the observations seen so far. This lets them use variable-length trajectories directly for RUL prediction.

3.4. Recurrent neural network

  • The RNN maps the current sensor input and prior hidden state to an updated state, enabling direct RUL prediction from sensor time series.This avoids reducing the sequence to a fixed-size engineered feature vector before prediction.Source: §3.4 · pp. 12–13
  • An RNN iterates over sensor sequences, updating hidden states that summarize the already-seen observations and pass information forward.The recurrent structure can process sequences of arbitrary length and use the complete sensor trajectory.Source: §3.4 · p. 13
  • The authors use a long short-term memory network because it can retain information over long sequences and is widely used in research and practice.For deep-network hyperparameter tuning, they use a random 10% sample of training data for validation.Source: §3.4 · p. 13

3.5. Proposed structured-effect neural network

The structured-effect neural network combines an explicit lifetime baseline with current sensor effects and a recurrent component that uses sensor history. Variational Bayesian estimation supplies uncertainty assessments for the component contributions.

3.5.1. Model specification

  • The model decomposes RUL into a non-parametric lifetime baseline, a covariate-based prediction, and a recurrent component incorporating sensor history.The structure assumes a population-wide general lifetime alongside sensor-based deviations that represent within-machine heterogeneity.Source: §3.5.1 · p. 14
  • The recurrent component increases flexibility over proportional-hazards models by using the full sensor sequence rather than only a covariate vector.The formulation also avoids an explicit need for feature engineering and trades off interpretability with flexible non-linear predictions.Source: §3.5.1 · p. 14
  • Practitioners can decompose predictions into population-wide baseline and machine-specific heterogeneity, with component coefficients indicating relative contributions.A negative recurrent contribution can indicate strong deterioration over time.Source: §3.5.1 · pp. 14–15
  • The lifetime baseline assumes conditional Weibull or log-normal distributions, while the linear component can use current readings or engineered features.The linear coefficients receive priors that regularize the model and can set some coefficients to zero for implicit variable selection.Source: §3.5.1 · p. 15

3.5.2. Model estimation through variational Bayesian inferences

  • Variational Bayesian estimation approximates parameter posteriors because MCMC runtime scales exponentially with parameter dimension and can become intractable for high-dimensional data.Unlike point estimates, variational Bayesian inferences quantify uncertainty, including uncertainty about the relative importance of model components.Source: §3.5.2 · p. 16

3.5.3. Estimation parameters

  • The experiments consider Weibull or log-normal priors for the non-parametric component, with parameters modeled using informative normal priors; linear-component coefficients use normal or Laplace priors.Source: §3.5.3 · p. 16

4. Computational experiments

The computational experiments assess predictive performance and interpretability on the Turbofan Engine Degradation Simulation benchmark, including uncertainty quantification and the training and performance of a staged model variation.

4.1. Dataset

  • The experiments used the Turbofan Engine Degradation Simulation dataset, with 200 engines, 21 sensors, and a predetermined 100-engine training and test split.Source: §4.1 · p. 17

4.2. Prediction performance for remaining useful life

  • The structured-effect model with a log-normal distribution and feature engineering improved on the best traditional model by 25.44%, with feature engineering accounting for 11.91% of the improvement.Source: §4.2 · p. 18
  • The LSTM achieved the lowest MAE, outperforming feature-engineered random forest by 37.12% and empirical RUL by 60.51%.Table 3 compares model specifications using MAE.Source: §4.2 · p. 18, Table 3 · p. 19

4.4. Estimated parameters

  • Variational Bayesian inference provides posterior uncertainty estimates for structural parameters, and the sensor coefficients’ confidence regions vary considerably, allowing standardized coefficients to rank sensor importance.The linear coefficients directly express how changes in sensor measurements affect the RUL forecast.Source: §4.4 · pp. 21–22

4.5. Model variations

  • Adding a weighting factor to the recurrent component produced inferior performance in all experiments; a two-stage estimation approach was also tested.The supplied passage reports no outcome for the two-stage approach.Source: §4.5 · p. 23
  • Although the staged approach was generally easier to train because each stage had fewer parameters, performance was inferior, possibly because staging prevented information sharing between components.The approach optimized the non-parametric and linear components first, then optimized the recurrent component against the residuals.Source: §4.5 · p. 23
  • 16.209 was the staged model variation’s mean absolute error, higher than for the proposed SENN.The passage reports that the staged approach’s mean absolute error increased to 16.209.Source: §4.5 · p. 23

5. Discussion

The discussion frames RUL prediction as support for maintenance decisions and presents structured-effect networks as combining interpretable lifetime and sensor components with deep-learning flexibility. It also considers domain knowledge, forecast interpretability, and limitations of variational Bayesian estimation.

5.1. Implication for decision support

  • Accurate RUL estimates can help companies plan maintenance ahead of time and prepare decision support, with predictive improvements potentially translating into operational cost savings.The discussion also says additional sensors describing machinery condition and environmental effects may improve forecast accuracy.Source: §5.1 · p. 24

5.2. Implications for the use of analytics

  • Encoding domain knowledge in a structural model offers an alternative to relying on vast input-data quantities, when the predicted variable’s approximate shape is known.The paper incorporates that shape through a probability density function and says variational inference can fit the predetermined structure.Source: §5.2 · p. 25

5.3. Implications from interpretable forecasts

  • Structured-effect networks combine nonlinear sensor modeling with interpretable forecasts, supporting conclusions about sensor relevance and possible machinery-design changes.The discussion gives a high moisture-sensor coefficient as a cue that designers might improve sealing.Source: §5.3 · pp. 25–26

5.4. Limitations and potential for future research

  • Dropout-based Bayesian approximation costs less to optimize, but it does not estimate coefficient uncertainty or incorporate coefficient priors.The authors choose variational Bayesian inference because prior information from experts is important in predictive maintenance.Source: §5.4 · p. 27

5.5. Concluding remarks

  • The structured-effect network outperforms traditional machine-learning models while supporting interpretable conclusions about deterioration sources.The authors describe it as combining accountability from distribution-based RUL models with a linear combination of sensor measurements and a recurrent component.Source: §5.5 · p. 27

A. Derivation of ELBO for structured-effect neural network

This appendix derives a tractable variational objective for approximating the model’s posterior. It relates fitting the variational distribution to minimizing its divergence from the posterior and maximizing an evidence lower bound.

A. Derivation of ELBO for structured-effect neural network

  • The variational distribution Qλ(θ) approximates the posterior P(θ | X), and the derivation frames fitting it in terms of Kullback-Leibler divergence.The optimal λ indexes the chosen variational distribution.Source: A. Derivation of ELBO for… · pp. 27–28
  • The derivation defines an optimal λ and corresponding variational distribution Qλ* as the target of the optimization.The appendix introduces a tractable approximation for this optimization.Source: A. Derivation of ELBO for… · p. 27
  • The marginal likelihood log P(X) is approximated by the evidence lower bound, providing a tractable objective for the variational approximation.The passage notes that the direct theorem is intractable because it depends on the model’s marginal likelihood.Source: A. Derivation of ELBO for… · p. 28

lower bound, ELBO(λ), i. e.,

The appendix turns the variational objective into an optimization procedure. It maximizes ELBO with gradient descent and estimates the objective and gradient using Monte Carlo integration.

lower bound, ELBO(λ), i. e.,

B. Two-stage estimation

The two-stage estimation specifies priors for the non-parametric and linear components, then estimates the recurrent component by maximizing its posterior given the data and fitted components.

B. Two-stage estimation

  • The non-parametric component λ follows a log-normal prior, with underlying parameters a and b modeled as normal priors centered at their empirical values with scale 1.Source: B. Two-stage estimation · p. 29
  • The linear coefficients β_i are assigned normal priors with mean 0 and scale 10.Source: B. Two-stage estimation · p. 29
  • Stage 2 estimates RNN parameters by maximizing their posterior conditional on X and the fitted λ* and β*.The recurrent component is implemented as a two-layer LSTM with 100 and 50 neurons.Source: B. Two-stage estimation · p. 29
Loading 1907.05146v2…