PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Learning Algorithms for Rotating Machinery Intelligent Diagnosis: An Open Source Benchmark Study

Zhibin Zhao, Tianfu Li, Jingyao Wu, Chuang Sun, Shibin Wang, Ruqiang Yan, Xuefeng Chen

arXiv:2003.03315v3eess.SPcs.LG

TL;DR

Rotating-machinery diagnosis lacks consistently fair evaluation because researchers use differing datasets, configurations, hyper-parameters, and rarely release comparison code. The paper benchmarks models and evaluation choices in a unified framework, finding frequency-domain input achieves the highest accuracy across datasets, while also releasing code and identifying open issues.

  • Problem

    Different datasets, inputs, hyper-parameters, and limited public code hinder fair comparison of deep-learning diagnosis algorithms.

  • Method

    The study evaluates deep-learning models across public datasets using varied input formats, normalization methods, data splits, augmentation methods, and a unified code framework.

  • Results

    Frequency-domain input achieves the highest accuracy across all datasets, and benchmark results expose dataset suitability and model overfitting concerns.

  • Takeaways & Limitations

    The released library and benchmark accuracy support fairer, quicker, and more reproducible comparisons of rotating-machinery diagnosis models.

  • Takeaways & Limitations

    The study identifies class imbalance, generalization ability, interpretability, few-shot learning, and model selection as unresolved issues.

Abstract

from arXiv · show

With the development of deep learning (DL) techniques, rotating machinery intelligent diagnosis has gone through tremendous progress with verified success and the classification accuracies of many DL-based intelligent diagnosis algorithms are tending to 100\%. However, different datasets, configurations, and hyper-parameters are often recommended to be used in performance verification for different types of models, and few open source codes are made public for evaluation and comparisons. Therefore, unfair comparisons and ineffective improvement may exist in rotating machinery intelligent diagnosis, which limits the advancement of this field. To address these issues, we perform an extensive evaluation of four kinds of models, including multi-layer perception (MLP), auto-encoder (AE), convolutional neural network (CNN), and recurrent neural network (RNN), with various datasets to provide a benchmark study within the same framework. We first gather most of the publicly available datasets and give the complete benchmark study of DL-based intelligent algorithms under two data split strategies, five input formats, three normalization methods, and four augmentation methods. Second, we integrate the whole evaluation codes into a code library and release this code library to the public for better development of this field. Third, we use specific-designed cases to point out the existing issues, including class imbalance, generalization ability, interpretability, few-shot learning, and model selection. By these works, we release a unified code framework for comparing and testing models fairly and quickly, emphasize the importance of open source codes, provide the baseline accuracy (a lower bound) to avoid useless improvement, and discuss potential future directions in this field. The code library is available at https://github.com/ZhaoZhibin/DL-based-Intelligent-Diagnosis-Benchmark.

1. Introduction

The paper addresses unreliable comparisons in rotating-machinery diagnosis by benchmarking diverse deep-learning choices within a unified framework. It reports dataset, input, splitting, augmentation, and model-selection findings while releasing reproducible code and discussing unresolved issues.

  • Rotating machinery operates under complex conditions, making efficient and precise fault diagnosis important because failures can cause accidents, economic loss, and casualties.
  • Different datasets, inputs, hyper-parameters, and limited code availability can produce unfair comparisons and ineffective algorithmic improvement.
  • The study collects nine public datasets and evaluates diagnosis algorithms on a unified platform using varied inputs, normalization, data splits, augmentation, and models.
  • Frequency-domain input achieves the highest accuracy across all datasets, while basic models already attain very high accuracy on datasets such as CWRU and XJTU-SY.
  • Order splitting may be more appropriate for sequential industrial data because random splitting can expose future patterns and produce virtually high accuracy through test leakage.
  • The released code library provides benchmark accuracy as a lower bound and supports fairer, quicker, and more reproducible comparisons of new models.
  • The paper examines class imbalance, generalization, interpretability, few-shot learning, and model selection as existing issues requiring further study.

2. Brief Review

The review surveys recent deep-learning research in rotating-machinery diagnosis, while positioning this work as a benchmark study rather than another review. It highlights diverse model-improvement strategies and the need for open-source, comparable evaluations.

  • Because existing reviews already cover the field, the paper reviews related 2020 publications to fill a stated void and focuses on benchmarking intelligent diagnosis.
  • Deep-learning publications on intelligent diagnosis increased year by year, alongside many review papers published in the recent four years.
  • Recent reviews cover AE, DBN, CNN, RNN, traditional machine learning, signal processing, transfer learning, and PHM applications across machinery domains.
  • Reported model-improvement approaches include preprocessing, ensemble learning, semi-supervised learning, GAN-based augmentation, regularization, transfer learning, and domain adaptation.
  • CNN studies additionally use varied input representations, wavelet regularization, information fusion, focal loss, transferable features, and relevance propagation.
  • The authors emphasize open-source codes and benchmark studies as priorities for further development.

3. Evaluation Algorithms

The benchmark evaluates MLP, AE, CNN, and RNN families using architectures adapted to one- and two-dimensional inputs. Their designs combine feature learning or reconstruction with classification, while BiLSTM represents the recurrent models.

  • The code library tests four model families—MLP, AE, CNN, and RNN—while excluding DBN because its training procedure differs substantially.
  • MLP: MLP uses five fully connected layers and five batch-normalization layers for one-dimensional input data.
  • AE: The benchmark designs fully connected AE variants for one-dimensional inputs and convolutional AE variants for two-dimensional inputs, with adaptive structures and hyper-parameters.
  • AE: DAE reconstructs clean inputs from noisy inputs, whereas SAE adds a sparsity constraint to the mean squared error loss.
  • AE: AE models learn low-dimensional features through jointly trained encoders and decoders, followed by encoder-classifier training with softmax cross-entropy.
  • CNN: The CNN benchmark includes five-layer 1D and 2D CNNs plus adapted LeNet, ResNet18, and AlexNet models.
  • CNN: CNNs use convolutional and max-pooling layers for feature learning before a classifier maps extracted high-level features into classes.
  • RNN: BiLSTM models are used for one- and two-dimensional inputs because they capture bidirectional long-distance dependencies and selectively remember or forget information.

4. Datasets

The study assembles publicly available labeled datasets for cross-model evaluation, ultimately using seven datasets for verification. Several run-to-failure datasets are excluded or restricted because their labels or failure categorization do not support the intended classification.

  • Dataset selection: The authors collected nine commonly used labeled datasets, excluding PHM 2012 and IMS from fault-classification verification because their labels were unsuitable.
  • Dataset selection: Seven datasets were ultimately used to verify the evaluated models.
  • CWRU: CWRU provides vibration signals from normal and single-point-fault bearings under four motor loads, classified into ten categories by fault size.
  • MFPT: MFPT includes baseline, outer-ring-fault, and inner-ring-fault bearing datasets sampled at 97656 Hz or 48828 Hz.
  • PU: PU contains current and vibration signals from 32 bearing sets across four working conditions, including undamaged, artificially damaged, and real-damage bearings.
  • PU: For PU verification, the study uses real-damaged bearings under condition N15 M07 F10, vibration signals only, and thirteen classes after deleting a duplicate bearing.
  • XJTU-SY: XJTU-SY contributes fifteen run-to-failure bearing classes, using data collected at the end of the experiments.
  • SEU: SEU contains bearing and gear sub-datasets from a drivetrain simulator, with twenty classes across two rotating-speed/load conditions.

5. Data Prepreocessing

The benchmark examines how five input formats and three normalization methods affect deep-learning diagnosis performance. It also adjusts two-dimensional input sizes to balance reconstruction difficulty for auto-encoders against feature-extraction difficulty for CNNs.

  • Five input formats and three normalization methods are evaluated because input type affects feature-extraction difficulty and normalization affects calculation difficulty.
  • Input formats: Time-domain inputs use unprocessed vibration signals, with 1024 points per sample and N samples obtained from the signal length.The sample count is determined by Eq. 9, where L is signal length and floor denotes rounding toward minus infinity.
  • Input formats: Frequency-domain inputs apply FFT to each time-domain sample, halve the data length, and retain the first half of the transform.
  • Input formats: Time-frequency and wavelet inputs use STFT and CWT, producing 33x33 and 100x100 representations, respectively.STFT uses a Hanning window of length 64, while CWT uses 100-point samples because it is time-consuming.
  • Input formats: Slicing-image inputs reshape samples into 32x32 images, with CNN-compatible sizes adapted through AdaptiveMaxPooling.Two-dimensional sizes are compromised across architectures: AE inputs are adjusted to 32x32, while CNN inputs use larger representations.
  • Input-size design: The preprocessing compromise addresses opposing architecture constraints: large 2D inputs hinder AE reconstruction, whereas small 2D inputs hinder CNN feature extraction.
  • Normalization: Three normalization methods are compared to facilitate subsequent processing and accelerate deep-learning convergence.The methods include Maximum-Minimum, [-1-1], and Z-score normalization.

6. Data Augmentation

The paper tests simple augmentation strategies because augmentation can diversify training data and alleviate small-sample learning difficulties, while label correctness remains domain-dependent. The authors characterize these strategies as a simple test whose applications require deeper study.

  • Data augmentation is intended to diversify training datasets and alleviate learning difficulties caused by small-sample problems.
  • Label-corrected augmentation is difficult because generated-sample validity depends mainly on domain knowledge.
  • The paper provides augmentation techniques to raise broader research attention, but describes their applications as insufficiently studied in depth.
  • One-dimensional strategies: RandomAddGaussian adds noise drawn from N(0, 0.01) to a one-dimensional input signal.
  • One-dimensional strategies: RandomScale multiplies one-dimensional signals by a factor following N(1, 0.01), while RandomStretch resamples signals and restores equal length.
  • One-dimensional strategies: RandomCrop masks a randomly positioned subsequence of one-dimensional signals, using length 10 in this paper.
  • Two-dimensional strategies: For two-dimensional inputs, RandomScale uses N(1, 0.01), and RandomCrop masks a randomly positioned subsequence of length 20.

7. Data Split

The benchmark contrasts random and order-based data splitting while emphasizing preprocessing without overlap to avoid test leakage. It uses validation for model selection and reports testing accuracy using a last-epoch procedure for convenience.

  • Random splitting is common, but overlapping preprocessing can cause test leakage when samples from training and testing data share processed information.
  • The formal model-selection procedure further divides the training set into training and validation subsets.
  • Evaluation procedure: For testing convenience and time saving, the study uses one-fold validation and treats last-epoch accuracy as testing accuracy, while also listing maximum accuracy.
  • Evaluation procedure: The study warns that selecting parameters using the validation set's maximum accuracy can be dangerous because it uses the validation data for selection.
  • Order splitting is more appropriate for industrial data because such data are typically sequential and may contain trends or temporal correlations.
  • Order split: Order splitting assigns earlier time phases to training and later time phases to testing, with a four-fold cross-validation condition also shown.

8. Evaluation Methodology

The evaluation methodology compares models under controlled data preparation, splitting, and training configurations. Performance is measured with overall accuracy across repeated experiments, using standardized setup choices to support comparison.

  • The methodology evaluates diagnosis algorithms with overall accuracy, defined as correctly classified samples divided by total samples.
  • Metrics: The paper distinguishes overall accuracy, average accuracy, and confusion matrices, but uses only overall accuracy for algorithm evaluation.
  • Metrics: Each experiment is repeated five times because deep-learning performance fluctuates during training; mean and maximum accuracy indicators are then reported.
  • Metrics: The indicators include mean and maximum overall accuracy at the last epoch, with last-epoch accuracy intended to represent real accuracy without test leakage.
  • Data preparation: The preparation stage compares random and order splits, using non-overlapping sliding windows and 1024-point samples.Random split uses 80% training and 20% testing samples, whereas order split uses the former 80% and latter 20% of the time series.
  • Experiment setup: Three experiment configurations test the effects of input type, normalization, and data splitting on model performance.
  • Training setup: Models are trained for 100 epochs with Adam, a learning rate of 0.001, and batch size 64.
  • Implementation: Experiments run under Windows 10 with PyTorch 1.1 on an Intel Core i7-9700K, GeForce RTX 2080Ti, and 16G RAM.

9. Evaluation Results

The evaluation compares models across multiple datasets and preprocessing choices, revealing dataset-specific differences in model, input, normalization, augmentation, and split performance.

  • CWRU Dataset: CNN models generally outperform AE models on CWRU, while FFT and STFT inputs improve accuracy across the tested input formats.
  • JNU Dataset: FFT inputs improve JNU performance, Z-score favors CNN models, -1-1 favors AE models, and augmentation improves both model types.
  • MFPT Dataset: FFT improves MFPT performance, with AE models outperforming CNN models; Z-score favors CNN, -1-1 favors AE, and augmentation improves both.
  • PU and SEU Datasets: For PU and SEU, FFT improves accuracy, Z-score benefits both model types, augmentation helps CNN and sometimes AE models, and order splits reduce performance.
  • UoC and XJTU-SY Datasets: UoC models generally perform poorly, whereas XJTU-SY models perform well; FFT or STFT, Z-score normalization, augmentation, and random splits generally improve results.

G. XJTU-SY Dataset

XJTU-SY is among the easier benchmark datasets: most models perform well, with frequency-domain inputs and Z-score normalization generally producing stronger accuracy.

  • Dataset Results: Most XJTU-SY models perform well, and CNN models generally achieve higher accuracy than AE models.
  • Input and Normalization: FFT and STFT processing improve XJTU-SY accuracy, while Z-score normalization benefits both AE and CNN models.
  • Augmentation and Splitting: Data augmentation improves final accuracy across different models, whereas order splitting slightly reduces accuracy.
  • Dataset Difficulty: Across seven datasets, CWRU, SEU, and XJTU-SY reach 100% accuracy, while UoC is much less accurate and the datasets separate into four difficulty levels.
  • Input Formats: Frequency-domain inputs achieve the highest accuracy across all datasets, followed by time-frequency inputs, because fault information is easier to distinguish in frequency representations.
  • Model Selection: ResNet18 performs best on most datasets, but AE models outperform other models on MFPT and UoC, indicating that greater complexity is not universally better.
  • Normalization: Normalization performance depends on the model and dataset, although Z-score normalization generally produces better accuracy.
  • Augmentation: Augmentation can slightly degrade already-high accuracy but improves lower-accuracy datasets, especially with time-domain inputs.

10. Discussion

The discussion uses targeted experiments to expose weaknesses in deep-learning diagnosis, including class imbalance, cross-condition generalization, interpretability, and few-shot learning.

  • Class Imbalance: The imbalance experiment constructs three groups with increasing imbalance ratios using PU bearing data, ResNet18, two input types, augmentation, and random splitting.
  • Class Imbalance: Highly imbalanced Group3 data yields much lower overall accuracy than balanced Group1 data, showing that class imbalance greatly degrades model performance.
  • Generalization Ability: The transfer experiment trains on one of three PU working conditions and tests on another across six train-test groups.
  • Generalization Ability: Models trained under one working condition generally perform poorly on another, indicating insufficient generalization ability.
  • Interpretability: High accuracy in interpretability experiments conflicts with the expectation that identical fault patterns and conditions should not be easily separated.
  • Interpretability: The results suggest models distinguish collection conditions rather than essential fault characteristics, motivating physically informed mechanisms.
  • Few-Shot Learning: Reducing samples from 100 to 1 lowers Best-Max accuracy from 91.46% to 20.39% for time-domain input and from 97.73% to 29.67% for frequency-domain input.
  • Few-Shot Learning: FFT can increase few-shot accuracy, but the resulting performance remains too low when sample numbers are greatly reduced.

11. Conclusion

The paper benchmarks four model families across nine public datasets, releases evaluation code, and uses benchmark results to support fairer comparisons and future research.

  • Contributions: The study evaluates MLP, AE, CNN, and RNN models across nine publicly available datasets from multiple perspectives.
  • Contributions: Benchmark accuracies provide lower-bound references intended to avoid useless improvement in future model development.
  • Open Source Code: The released code library enables researchers to test their own diagnosis models on the benchmark datasets.
  • Implications: The evaluation results and code library are intended to promote understanding of DL models and provide a unified framework for developing more effective models.
  • Future Work: Future work focuses on class imbalance, generalization ability, interpretability, few-shot learning, and model selection.

Appendix A. Evaluation Results

The appendix organizes evaluation results across several model families, input representations, normalization or location settings, and reporting epochs. It also defines five input formats and notes when samples are reshaped into two-dimensional matrices.

  • Model coverage: Results are organized for AE, DAE, SAE, MLP, CNN, LeNet, AlexNet, ResNet18, and LSTM models.
  • Input formats: Five input formats are identified: time domain, frequency domain, wavelet domain, time-domain samples after STFT, and time-domain samples.
  • Input processing: Some inputs are reshaped into a 2D matrix before evaluation.
  • Epoch reporting: For the same input, the first result line represents the last epoch and the second represents the best epoch.
  • Result layout: The reported result layouts distinguish normalization or location settings and paired Mean and Max values.
Loading 2003.03315v3…