PaperlayerView exact revision on arXiv

Source-linked AI summary

Dual Aspect Self-Attention based on Transformer for Remaining Useful Life Prediction

Zhizheng Zhang, Wen Song, Qiqiang Li

arXiv:2106.15842v3eess.SPcs.AIcs.LG

TL;DR

RUL prediction is important for condition-based maintenance, but existing methods face difficulties with long sequences and with extracting information across sensors and time steps. DAST uses parallel sensor and time-step self-attention encoders in a Transformer encoder-decoder without RNN/CNN modules, and experiments on two turbofan datasets report superior performance to state-of-the-art methods.

  • Problem

    Existing RUL prediction methods have difficulty processing long sequences and extracting important information from both sensor and time-step aspects.

  • Method

    DAST is a Transformer encoder-decoder that uses parallel sensor and time-step self-attention encoders without RNN or CNN modules.

  • Results

    DAST significantly outperforms state-of-the-art RUL prediction methods on two widely used turbofan engine datasets.

  • Takeaways & Limitations

    DAST adaptively learns the importance of sensors and time steps while avoiding mutual influence between the two feature-extraction aspects.

Abstract

from arXiv · show

Remaining useful life prediction (RUL) is one of the key technologies of condition-based maintenance, which is important to maintain the reliability and safety of industrial equipments. Massive industrial measurement data has effectively improved the performance of the data-driven based RUL prediction method. While deep learning has achieved great success in RUL prediction, existing methods have difficulties in processing long sequences and extracting information from the sensor and time step aspects. In this paper, we propose Dual Aspect Self-attention based on Transformer (DAST), a novel deep RUL prediction method, which is an encoder-decoder structure purely based on self-attention without any RNN/CNN module. DAST consists of two encoders, which work in parallel to simultaneously extract features of different sensors and time steps. Solely based on self-attention, the DAST encoders are more effective in processing long data sequences, and are capable of adaptively learning to focus on more important parts of input. Moreover, the parallel feature extraction design avoids mutual influence of information from two aspects. Experiments on two widely used turbofan engines datasets show that our method significantly outperforms the state-of-the-art RUL prediction methods.

I. INTRODUCTION

RUL prediction supports condition-based maintenance, but existing approaches struggle with complex equipment, long sequences, and jointly modeling sensor and time-step importance. DAST addresses these issues with parallel self-attention encoders within a Transformer-based RUL predictor.

  • RUL prediction is central to condition-based maintenance because it supports maintenance decisions using equipment health and failure-time information.
  • Accurate model-based RUL prediction is difficult because modern industrial equipment contains complex nonlinear relationships across systems and components.
  • RNN methods can forget important information and learn long-term dependencies less effectively because sequence data passes through recurrent units sequentially.
  • CNN methods have limited long-term dependency capture because processing longer sequences requires progressively larger convolution kernels and receptive fields.
  • Attention-based RNN/CNN methods retain long-sequence inefficiency and can create mutual influence because attention and feature-extraction modules process inputs sequentially.
  • DAST uses parallel sensor and time-step self-attention encoders without RNN/CNN modules, fuses their features, and produces RUL predictions through a self-attention decoder.

III. METHODOLOGY

The paper formulates RUL prediction as learning a mapping from turbofan-engine condition-monitoring data to real-time RUL outputs.

  • RUL prediction uses sensor-collected condition-monitoring data from turbofan-engine operation as input.
  • The input is a time sequence X_t ∈ R^k with T time steps and k sensors, while the output is predicted RUL Y_t at each time step.
  • DAST establishes the mapping between condition-monitoring data and real-time RUL through a Transformer-based deep architecture.

B. Model Architecture

DAST processes condition-monitoring sequences through parallel sensor and time-step self-attention encoders, then fuses their representations into a joint feature map.

  • Encoder structure: The encoder includes input embedding, positional encoding, stacked sensor encoder layers, and stacked time-step encoder layers.Input embedding maps monitoring data to D_model-dimensional vectors, while positional encoding supplies sequence-position information.
  • Sensor encoder: The sensor encoder applies multi-head self-attention along the sensor dimension to learn which sensor features receive higher weights.Queries, Keys, and Values are generated from position-encoded inputs, and softmax produces sensor weights.
  • Time-step encoder: The time-step encoder has the same general structure but extracts features along time, allowing attention to more important steps for RUL prediction.It forms time-step Queries, Keys, and Values and obtains time-step weights through softmax along the time dimension.
  • Feature fusion: Feature fusion combines sensor features F_s ∈ R^(dk×D_model) and time-step features F_t ∈ R^(T×D_model) into a new feature map.A trainable matrix W_f is used to capture information from both representations.
  • Dual-aspect encoding: DAST uses parallel sensor and time-step encoders to extract importance-weighted features from both dimensions simultaneously.The parallel arrangement is intended to avoid mutual influence between the two feature-extraction aspects.

D. Decoder of DAST

The DAST decoder follows the original Transformer decoder pattern to transform encoder representations into RUL predictions.

  • The decoder contains an input embedding layer, stacked decoder layers, a flatten layer, and a feed-forward network.Each decoder layer includes masked multi-head self-attention and encoder-decoder multi-head self-attention.

IV. EXPERIMENTS

The experiments evaluate DAST against state-of-the-art RUL prediction methods using two widely used turbofan-engine datasets and sliding-window processing.

  • The experimental study uses two widely used turbofan-engine datasets to evaluate DAST against state-of-the-art RUL prediction methods.
  • Sliding-window processing is included as part of the experimental setup for the RUL prediction evaluation.

A. Datasets

The C-MAPSS dataset comprises four turbofan-engine sub-datasets with differing operating conditions and fault modes. F002 and F004 are more difficult RUL prediction settings than F001 and F003.

  • C-MAPSS contains four sub-datasets: FD001, FD002, FD003, and FD004.
  • The sub-datasets differ in their numbers of operating conditions and fault modes.
  • F002 and F004 have more complex operating conditions and fault modes, making their RUL harder to predict.

B. Experimental Setting

The experimental pipeline normalizes sensor data, segments multivariate sequences with sliding windows, and augments the sequence with statistical features.

  • Min-max scaling normalizes sensor data with different units and scales before prediction.The procedure uses each sequence's minimum and maximum values.
  • Sliding windows segment multivariate time series, using the last point's RUL as the label for each window.The window stride is one, and the window length is denoted by T_w.
  • The model receives sequence statistics consisting of the mean value and regression coefficient estimates.

2) Evaluation metrics:

Evaluation uses RMSE and Score, while the reported configuration is selected through grid search and trained with RMSE loss.

  • Evaluation metrics: RMSE and Score are the two performance indicators used to evaluate prediction effectiveness.
  • Evaluation metrics: The Score metric penalizes overestimated RUL more heavily than underestimated RUL because optimistic predictions can have more serious practical consequences.
  • Evaluation metrics: Lower values indicate better prediction accuracy for both RMSE and Score.
  • Implementation: Grid search selects the DAST configuration, which is trained for 100 epochs with RMSE as the loss function.The learning rate is 0.001 and the batch size is 256.

1) Comparison on the C-MAPSS dataset:

DAST is compared with established RUL predictors on C-MAPSS and shows stronger overall accuracy, favorable engine-level comparisons, and real-time testing speed. Its gains are especially pronounced on the more difficult datasets and on Score.

  • Comparison on the C-MAPSS dataset: DAST achieves the smallest average RMSE and Score among the compared methods across the C-MAPSS sub-datasets.The comparison includes pure RNN/CNN methods, attention-based methods, and health-indicator-based methods.
  • Comparison on the C-MAPSS dataset: 10.02% lower RMSE on F002 and 35.67% lower Score on F004 are reported relative to the best existing methods.
  • Detailed analysis of the prediction results: DAST predictions follow actual RUL trajectories more closely than BiLSTM and DCNN on randomly selected engines.Most DAST predictions are close to or below actual RUL, which is desirable because overestimation has more serious consequences.
  • Detailed analysis of the prediction results: On F002's 259 engines, DAST has lower RMSE and Score than BiLSTM on 75.6% and 72.6% of engines, respectively.
  • Detailed analysis of the prediction results: On the same F002 comparison, DAST outperforms DCNN on RMSE and Score for 78.3% and 81.6% of engines, respectively.
  • Efficiency: Testing 100 engines takes 0.03s on a CPU, or 0.3ms per engine, meeting the stated requirement for real-time RUL prediction.

3) Comparison on the PHM 2008 dataset:

On PHM 2008, DAST is evaluated against recent RUL prediction methods and is reported to outperform all baselines. The experiments also examine time-window effects on C-MAPSS.

  • PHM 2008 comparison: DAST significantly outperforms all listed baselines on the PHM 2008 dataset.The comparison uses a sliding time window of Tw = 60 and evaluates predictions with the Score metric obtained from the NASA repository.
  • PHM 2008 comparison: 46.7% improvement over the most competitive method in is reported for PHM 2008.
  • Time-window analysis: RMSE and Score are smallest at time-window length 40 on the F001 and F003 C-MAPSS datasets.The study varies Tw from 30 through 70 to assess the parameter’s influence.
  • Time-window analysis: Larger time windows could contain more degradation information on the F001 and F003 datasets.

2) Ablation study of DAST:

DAST’s design separates sensor and time-step feature extraction through parallel self-attention encoders. The paper also visualizes learned weights to show which inputs are more informative for RUL prediction.

  • Learned weights: For engine unit 99 in F003, sensors T50, Nc, and phi are more important, while time steps 15–24 and the last step are more informative.
  • Architecture: Self-attention processes CBM data points without considering their distance, supporting long-term dependency capture.
  • Ablation study: The parallel encoding design is reported to outperform the vanilla Transformer structure in ablation experiments.
  • Architecture: DAST uses parallel sensor and time-step encoders to extract weighted features from both aspects simultaneously.The two feature sets are subsequently fused, avoiding mutual influence between sensor and time-step information.
  • Learned weights: DAST adaptively learns the importance weights of different sensors and time steps without human intervention.These weights can be interpreted by maintenance personnel to focus on more important inputs in real time.
Loading 2106.15842v3…