Deep Learning Algorithms for Rotating Machinery Intelligent Diagnosis: An Open Source Benchmark Study

• Order-based splits are more appropriate for sequential industrial data because random splits can leak overlapping or future patterns into evaluation.The introduction notes that random splitting can cause test leakage when preparation overlaps, and may let algorithms record future patterns.Source: §1 · p. 2
• The study gathers nine public datasets, evaluates algorithms on seven, and compares data preparation and deep-learning methods within a unified benchmark.The evaluation varies input formats, normalization, data splits, and augmentation; the authors also discuss dataset adaptability and suitability.Source: §1 · pp. 2–3
• Frequency-domain input achieved the highest accuracy across the datasets, while random splits may produce virtually high accuracy and order-based splits may be more suitable.The paper also warns that basic models can already achieve very high accuracy on CWRU and XJTU-SY, limiting their usefulness for comparing methods.Source: §1 · p. 2
• CNNs do not achieve the best results in every case, and their use should account for overfitting.When dataset accuracy is not high, augmentation improves model performance, especially with time-domain input.Source: §1 · p. 2
• The authors release a unified code library and benchmark accuracies as lower bounds to support reproducible, fairer comparisons and avoid useless improvement.The library is intended to enable better comparisons and is described as an open-source contribution.Source: §1 · pp. 2–3
• The paper identifies class imbalance, generalization, interpretability, few-shot learning, and model selection as issues for further study.It argues that inconsistent recommended inputs and hyperparameters, together with limited public code, can lead to unfair comparisons and ineffective improvement.Source: §1 · pp. 2–3

Publication trend

• The literature search found that papers on deep-learning diagnosis algorithms increased year by year, while many related review papers appeared in the four years before the study.Figure 1 describes the publication search and covers the six years through April 2020.Source: §2 · p. 3, Figure 1 · p. 4

Review scope

• The paper summarizes existing review papers for newcomers and surveys related work published in 2020 to extend coverage beyond reviews published before that year.The authors state that the 2020 survey is intended to fill a void in the existing review coverage.Source: §2 · pp. 3–5

AE research directions

• For improving AE models, studies combined them with singular value decomposition and nonlinear frequency spectrum, ensemble learning, semi-supervised methods, and pretrained, fine-tuned transfer.Source: §2 · p. 5

CNN research directions

• CNN models used images, infrared thermal images, and two-dimensional images as inputs; for imbalanced learning, GANs generated new labeled samples, and focal loss addressed severe imbalance.Source: §2 · p. 5

Gap and motivation

• Few studies comprehensively evaluate deep-learning diagnosis algorithms across publicly available datasets, provide benchmark accuracy, and release complete evaluation code; this paper aims to fill that gap.The authors note that earlier code and accuracy comparisons were not comprehensive across datasets, models, or configurations.Source: §2 · p. 5

3. Evaluation Algorithms

• The benchmark evaluates MLP, AE, CNN, and RNN models; DBN is excluded because its training procedure differs from those four models.The authors note that the absence of an open-source community makes it impossible to cover all published models.Source: §3 · pp. 5–6

3.1. MLP

• For 1D inputs, the MLP uses five fully connected layers and five batch-normalization layers, while its large parameter count can make complex tasks hard to train.The paper identifies MLP as a simple network suited to easier classification tasks.Source: §3.1 · p. 6

3.2. AE

• The AE variants use fully connected layers for 1D inputs and convolutional layers for 2D inputs, with DAE and SAE changing the input or loss.DAE reconstructs clean input from noise-corrupted input, whereas SAE adds a sparsity constraint to MSE loss; the encoder-decoder learns features before the encoder and classifier are jointly trained for classification.Source: §3.2 · p. 7

3.3. CNN

• The study designs five-layer 1D and 2D CNNs and adapts LeNet, ResNet18, and AlexNet to both input types.Convolutional and max-pooling layers learn features, which a fully connected classifier maps to classes.Source: §3.3 · p. 8

3.4. RNN

• The study uses BiLSTM for 1D and 2D inputs, leveraging forward and backward LSTM blocks to capture bidirectional dependencies across time series.RNN training can suffer from vanishing and exploding gradients; LSTM gates selectively manage forgetting, updating, and outputting information.Source: §3.4 · pp. 8, 10

4.1. CWRU Bearing Dataset

• CWRU drive-end signals sampled at 12 kHz were grouped into ten categories: one healthy state and nine faults spanning three fault types and sizes.The data covered four motor loads, with signals originally collected at 12 kHz or 48 kHz.Source: §4.1 · pp. 10–11

4.2. MFPT Bearing Dataset

• MFPT bearing data were classified into 15 categories: one healthy state and 14 inner-ring or rolling-element fault states distinguished by load.The datasets included baseline recordings sampled at 97656 Hz and fault recordings sampled at 48828 Hz.Source: §4.2 · p. 11

4.3. PU Bearing Dataset

• For PU evaluation, the authors used vibration signals from real-damaged bearings under condition N15 M07 F10, yielding 13 classes after removing a duplicate bearing.The source dataset comprised 32 sets of current and vibration signals, including undamaged, artificially damaged, and real-damaged bearings.Source: §4.3 · pp. 11–13

4.4. UoC Gear Fault Dataset

• UoC gear signals sampled at 20 kHz covered nine gear conditions, classified as one healthy state and eight fault states.Fault types included missing tooth, root crack, spalling, and chipping tip at five severity levels.Source: §4.4 · p. 13

4.5. XJTU-SY Bearing Dataset

• XJTU-SY evaluation used all described end-of-run-to-failure data from 15 bearings across three working conditions, with 15 classes in total.Signals were collected at 2.56 kHz, with 32768 data points recorded at one-minute intervals.Source: §4.5 · p. 13

4.6. SEU Gearbox Dataset

• SEU gearbox data comprised 20 classes across two speed-load conditions, and the study used the second row of vibration signals from each file.The conditions were 20 Hz - 0 V and 30 HZ - 2 V.Source: §4.6 · pp. 13–14

4.7. JNU Bearing Dataset

• JNU bearing signals sampled at 50 kHz covered three rotating speeds and 12 classes combining one healthy state with three fault modes.The fault modes were inner-ring, outer-ring, and rolling-element faults.Source: §4.7 · p. 14

4.8. PHM 2012 Bearing Dataset; 4.9. IMS Bearing Dataset

• Because PHM 2012 provided no failure-type labels, it was not used; IMS failures occurred at different bearing locations, making three-class classification inappropriate, and its datasets were not evaluated.Source: §4.8 · p. 14, §4.9 · p. 14

5.1. Input Types

• Which of five input types best suits intelligent diagnosis remains an open question, so the study compares their effects on model performance.Source: §5.1 · p. 15

5.1.1. Time Domain Input

• Each 1024-point time-domain sample is used without preprocessing, with 80% of samples assigned to training and 20% to testing.Source: §5.1.1 · p. 15

5.1.2. Frequency Domain Input

• FFT transforms each sample to the frequency domain and retains the first half of the result, halving the data length.Source: §5.1.2 · p. 15

5.1.3. Time-Frequency Domain Input

• STFT with a Hanning window of length 64 produces a 33x33 time-frequency representation for each sample.Source: §5.1.3 · p. 15

5.1.4. Wavelet Domain Input

• Because CWT is time-consuming, samples are shortened to length 100, yielding 100x100 wavelet-coefficient images.Source: §5.1.4 · pp. 15–16

5.1.5. Slicing Image Input

• Slicing reshapes each sample to 32x32; large 2D AE inputs hinder reconstruction and small CNN inputs impair feature extraction, so CNN CWT/STFT/slicing inputs are resized to 300x300/330x330/320x320.Source: §5.1.5 · p. 16

5.2. Normalization

• The study assesses maximum-minimum, [-1-1], and Z-score normalization for their effects on deep-learning model performance.Source: §5.2 · p. 16

6. Data Augmentation

• Augmentation can diversify training data and ease small-sample learning, but creating label-correct samples depends on domain knowledge and is difficult to verify; these techniques are preliminary tests needing deeper study.The paper offers the techniques to encourage further research into data augmentation for intelligent diagnosis.Source: §6 · p. 17

6.1. One Dimension Input Augmentation

• For one-dimensional signals, RandomAddGaussian adds noise drawn from N(0, 0.01), while RandomScale multiplies the signal by a factor drawn from N(1, 0.01).Source: §6.1 · p. 17
• For one-dimensional signals, RandomStretch resamples to a random proportion and restores equal length by nulling, while RandomCrop masks a randomly positioned subsequence of length 10.Source: §6.1 · p. 17

6.2. Two Dimension Input Augmentation

• For two-dimensional inputs, the described augmentations are RandomScale with a factor from N(1, 0.01) and RandomCrop, which masks a randomly positioned subsequence of length 20.Source: §6.2 · p. 17

Data Split

• Random splitting is common, but preprocessing must avoid overlap because overlapping sample preparation can leak test information into evaluation.Splitting training and testing data at the beginning of preprocessing allows processing to be applied to both sets without overlap.Source: §7 · p. 18
• The training set should be divided further into training and validation sets for model selection; selecting parameters by maximum validation accuracy is also described as dangerous.The validation set is used to select parameters, so choosing its maximum accuracy risks using that set for both selection and evaluation.Source: §7 · p. 18
• When no testing set is available, average accuracy across four-fold cross-validation can represent generalization accuracy; this study instead uses one-fold validation and last-epoch accuracy for testing, listing maximum epoch accuracy for comparison.The section notes that some papers report maximum validation accuracy, which is risky because validation data are used for parameter selection.Source: §7 · p. 18
• For industrial data that is sequential or temporally correlated, the section recommends splitting by time phase rather than randomly.The study compares this order-split strategy with random splitting and illustrates time-based four-fold cross-validation.Source: §7 · p. 18

8.1. Evaluation Metrics

• Only overall accuracy is used from the three standard metrics—overall accuracy, average accuracy, and confusion matrix—and it is the correctly classified sample count divided by total samples.The study repeats each experiment five times and reports mean and maximum overall accuracy at the last epoch and at the maximal-accuracy point.Source: §8.1 · pp. 18–19
• Each experiment is repeated five times, with mean and maximum overall accuracy summarized for both the last epoch and the maximal-accuracy point.The passage names the last-epoch indicators Last-Mean and Last-Max.Source: §8.1 · p. 19

8.2. Experimental Setting

• Random split assigns 80% of non-overlapping 1024-point windows to training and 20% to testing, whereas order split uses the first 80% of each time series for training and the last 20% for testing.For order split, non-overlapping 1024-point windows are then extracted from the training and testing time series.Source: §8.2 · p. 20
• Experiments examine input types, normalization, and split methods using Adam, a 0.001 learning rate, batch size 64, and 100 training epochs with testing interleaved.The three configurations are shown in Tables 8–10.Source: §8.2 · p. 20

9. Evaluation Results

• The evaluation summarizes results by dataset, input type, model, normalization, augmentation, and data split; Appendix A bolds accuracies above 95%.Source: §9 · p. 21

A. CWRU Dataset

• CNNs generally outperform AEs on CWRU; FFT/STFT inputs and CNN Z-score or AE -1-1 normalization improve accuracy, augmentation helps CNNs but not AEs, and order splitting slightly reduces accuracy.Source: A. CWRU Dataset · p. 21

B. JNU Dataset

• FFT inputs improve JNU accuracy; CNN Z-score and AE -1-1 normalization perform better, augmentation benefits both model types, and order splitting highly reduces accuracy.Source: B. JNU Dataset · p. 22

C. MFPT Dataset

• Time- or wavelet-domain inputs perform worse than FFT, which improves accuracy; AE accuracy exceeds CNNs in this dataset, Z-score favors CNNs, -1-1 favors AEs, augmentation improves both, and order splitting heavily reduces accuracy.Source: C. MFPT Dataset · p. 22

D. PU Dataset

• CNNs generally outperform AEs; wavelet inputs worsen PU accuracy while FFT/STFT improve it, Z-score benefits both model types, augmentation helps CNNs but not AEs, and order splitting heavily reduces accuracy.Source: D. PU Dataset · p. 23

E. SEU Dataset

• Time- and wavelet-domain inputs yield worse SEU accuracy, whereas FFT improves it and AEs outperform CNNs; Z-score and augmentation benefit both, while order splitting slightly reduces accuracy.Source: E. SEU Dataset · p. 23

F. UoC Dataset

• UoC is difficult for most models, with AlexNet relatively worse; FFT improves accuracy, AE generally outperforms CNN, Z-score helps AE and CNN, and augmentation helps while order split sharply reduces accuracy.Source: F. UoC Dataset · p. 23

G. XJTU-SY Dataset

• Most models perform well on XJTU-SY; FFT and STFT improve accuracy, CNN generally outperforms AE, Z-score and augmentation help, and order split slightly reduces accuracy.Source: G. XJTU-SY Dataset · p. 23

9.2. Results of Datasets

• Except for UoC, AE and CNN accuracy exceeds 95% on the evaluated datasets; CWRU, SEU, and XJTU-SY reach 100%, while UoC accuracy is much lower.The datasets are divided into four difficulty levels based on how often accuracy exceeds 95%.Source: §9.2 · p. 23

9.3. Results of Input Types

• Frequency-domain input achieves the highest accuracy across datasets, followed by time-frequency input; short-sample CWT processing and upsampling may degrade its accuracy.The paper attributes frequency-domain effectiveness to noise being spread across the band, making fault information easier to distinguish than in the time domain.Source: §9.3 · pp. 23–24

9.4. Results of Models

• ResNet18 performs best on most datasets, including CWRU, JNU, PU, SEU, and XJTU-SY, whereas AE models do better on MFPT and UoC, possibly due to dataset size and overfitting.The results caution that a more complex model does not perform better on every dataset.Source: §9.4 · p. 24

9.5. Results of Data Normalization

• No normalization method is universally best because results depend on the model and dataset, although Z-score normalization generally improves accuracy.Source: §9.5 · p. 24

9.6. Results of Data Augmentation

• Augmentation may slightly degrade results when datasets already achieve high accuracy, but improves lower-accuracy datasets, especially with time-domain input.The paper notes that augmentation can change the original data distribution and make learning harder.Source: §9.6 · p. 24

9.7. Results of Data Splitting

• Order splits yield similar results to random splits on CWRU and XJTU-SY but sharply reduce accuracy on PU and UoC; they may help avoid inflated random-split results, alongside the released benchmark library and lower-bound accuracy.The authors also discuss class imbalance, generalization, interpretability, few-shot learning, and model selection.Source: §9.7 · pp. 24–25

10.1. Class Imbalance

• Group3’s overall accuracy is much lower than Group1’s, indicating that class imbalance can greatly degrade model performance.The authors identify methods based on physical meaning and fault attention as areas for further exploration.Source: Figure 11 · p. 26, §10.1 · p. 26

10.2. Generalization ability

• In most cases, models trained on one working condition do not perform well on another, revealing insufficient generalization ability.The authors call for methods that transfer trained algorithms effectively across changing working conditions.Source: Figure 12 · p. 27, §10.2 · p. 27

10.3. Interpretability

• High accuracy on tasks with the same fault pattern and working condition across classes suggests models may learn collection-point differences, not essential fault characteristics.The authors argue that determining whether models learn essential fault features or merely classify collection conditions is important.Source: Figure 13 · p. 28, §10.3 · p. 28

10.4. Few-Shot Learning

• Best-Max accuracy falls from 91.46% to 20.39% for time-domain input and from 97.73% to 29.67% for frequency-domain input as samples per class drop from 100 to 1.Accuracy decreases sharply with fewer samples; even with FFT, performance remains too low to be accepted when sample counts are extremely small.Source: Figure 14 · p. 29, §10.4 · p. 29

10.5. Model selection

• The benchmark offers guidelines for choosing a baseline model to reduce costly trial and error in network design.The discussion also describes neural architecture search: a physically informed search space can yield dataset-specific networks through reinforcement learning, evolutionary algorithms, or gradient strategies, without manual construction.Source: §10.5 · p. 30

• The study evaluates MLP, AE, CNN, and RNN models on nine publicly available datasets from multiple perspectives and provides benchmark accuracy as a lower bound to avoid useless improvement.Source: §11 · p. 30
• The authors release a code library for testing researchers’ own models on these datasets and seek to support better model understanding and a unified framework for generating more effective models.Source: §11 · p. 30
• Future studies will address class imbalance, generalization ability, interpretability, few-shot learning, and model selection.Source: §11 · p. 30

Appendix A. Evaluation Results

• The numbered input labels identify time-domain, frequency-domain, wavelet-domain, and time-domain-after-STFT samples.The passage also labels input 5 as a time-domain sample without further qualification.Source: Appendix A · p. 36
• Some evaluation inputs are reshaped to a 2D matrix.Source: Appendix A · p. 36
• For the same input, the first line gives last-epoch results and the second line gives best-epoch results.Source: Appendix A · p. 36