PaperlayerView exact revision on arXiv

Source-linked AI summary

A Cost-Sensitive Deep Belief Network for Imbalanced Classification

Chong Zhang, Kay Chen Tan, Haizhou Li, Geok Soon Hong

arXiv:1804.10801v2cs.LGstat.ML

TL;DR

Imbalanced data challenge conventional DBNs because equal-cost assumptions can favor majority classes, while the appropriate misclassification costs are often unknown. ECS-DBN uses adaptive differential evolution to optimize class-dependent costs from training data before applying them to DBN classification. Across 58 benchmark datasets and a real-world fault-diagnosis dataset, the method is reported to outperform competing techniques, particularly on G-mean.

  • Problem

    Conventional DBNs assume equal class costs, while misclassification costs in imbalanced real-world problems are often unknown and cost-sensitive DBN learning remains insufficiently studied.

  • Method

    ECS-DBN uses adaptive differential evolution to optimize class-dependent misclassification costs from training data and then incorporates them into DBN classification.

  • Results

    ECS-DBN outperforms competing techniques on 58 benchmark datasets and a real-world fault-diagnosis dataset, with especially strong performance in G-mean.

  • Takeaways & Limitations

    The approach provides an evolutionary cost-sensitive strategy for handling unknown misclassification costs in imbalanced classification.

Abstract

from arXiv · show

Imbalanced data with a skewed class distribution are common in many real-world applications. Deep Belief Network (DBN) is a machine learning technique that is effective in classification tasks. However, conventional DBN does not work well for imbalanced data classification because it assumes equal costs for each class. To deal with this problem, cost-sensitive approaches assign different misclassification costs for different classes without disrupting the true data sample distributions. However, due to lack of prior knowledge, the misclassification costs are usually unknown and hard to choose in practice. Moreover, it has not been well studied as to how cost-sensitive learning could improve DBN performance on imbalanced data problems. This paper proposes an evolutionary cost-sensitive deep belief network (ECS-DBN) for imbalanced classification. ECS-DBN uses adaptive differential evolution to optimize the misclassification costs based on training data, that presents an effective approach to incorporating the evaluation measure (i.e. G-mean) into the objective function. We first optimize the misclassification costs, then apply them to deep belief network. Adaptive differential evolution optimization is implemented as the optimization algorithm that automatically updates its corresponding parameters without the need of prior domain knowledge. The experiments have shown that the proposed approach consistently outperforms the state-of-the-art on both benchmark datasets and real-world dataset for fault diagnosis in tool condition monitoring.

I. INTRODUCTION

Imbalanced classification is difficult because conventional methods favor majority classes under equal-cost assumptions. The paper focuses on cost-sensitive learning and evolutionary optimization to improve DBN classification without prior cost knowledge.

  • Skewed class distributions occur across applications including fault diagnosis, fraud detection, medical diagnosis, and tool condition monitoring.
  • Traditional learning algorithms can favor the majority class because they assume equal misclassification costs or balanced class distributions.
  • Resampling methods can alter the original class distribution, omit useful data, or add computational burden through redundant samples.
  • Cost-sensitive learning assigns differential costs to classes while addressing unequal error costs at the algorithmic level.
  • Misclassification costs are often unknown, motivating adaptive differential evolution to optimize class-dependent costs without prior domain knowledge.

A. Cost-sensitive Learning

Cost-sensitive learning assigns differential misclassification costs, while evolutionary methods can optimize those costs for imbalanced classification. The paper applies this perspective to DBNs, whose layered RBM architecture is pretrained and then fine-tuned for prediction.

  • A. Cost-sensitive Learning: Cost-sensitive learning assigns differential misclassification costs to the classes involved in a classification task.
  • A. Cost-sensitive Learning: Existing approaches such as WELM, CSMLP, and OPAL address imbalance but generally require user needs, expert knowledge, or prior knowledge to set costs.
  • B. Evolutionary Algorithm (EA): Evolutionary algorithms can optimize datasets or classifiers, but prior work largely focused on traditional simple networks rather than cost-sensitive deep learning.
  • A. Deep Belief Network: A DBN stacks Restricted Boltzmann Machines, connects contiguous layers, and learns progressively higher-level feature representations.
  • A. Deep Belief Network: DBN training uses greedy layer-wise unsupervised pretraining followed by supervised fine-tuning of the whole network.
  • A. Deep Belief Network: The DBN transforms inputs through hidden layers before a softmax output layer estimates class probabilities for prediction.

B. Cost-sensitive Deep Belief Network

Cost-sensitive DBN classification assigns class-dependent penalties and selects predictions by minimizing expected cost. The paper addresses unknown costs by optimizing them with adaptive differential evolution while applying cost sensitivity at the DBN output layer.

  • Cost-sensitive decision framework: Cost-sensitive learning minimizes overall training cost by assigning non-negative penalties to misclassification errors.The method models class-dependent error costs rather than assuming every error has the same consequence.
  • Cost-sensitive decision framework: For a sample x, the decision rule selects the class with minimum expected cost using posterior class probabilities and misclassification costs.The posterior probability P(j|x) contributes to the expected risk R(i|x) for predicting class i.
  • Cost-sensitive decision framework: Misclassification thresholds transform posterior probabilities into labels so that the resulting decisions minimize misclassification costs.The threshold value 1 − C_i,j is applied to the posterior probability P(y = j|x).
  • Cost-sensitive DBN: Traditional training assumes uniform class distributions and equal costs, whereas imbalanced problems require output-layer costs that reflect non-uniform class distributions.The proposed cost-sensitive method changes only the DBN output layer while retaining the stated pre-training and fine-tuning procedures.
  • Cost optimization: Adaptive differential evolution replaces trial-and-error cost selection by evolving candidate costs without prior domain knowledge.This addresses the practical problem that class costs are unknown and may vary across classes, including rare but important classes.

IV. EVOLUTIONARY COST-SENSITIVE DEEP BELIEF NETWORK (ECS-DBN)

ECS-DBN incorporates class-dependent misclassification costs directly into DBN classification and optimizes those costs through adaptive differential evolution. Training initializes candidate costs, evaluates cost-sensitive predictions on training data, and selects suitable costs for the model.

  • ECS-DBN approach: ECS-DBN incorporates a cost-sensitive function into the DBN classification paradigm while optimizing class-dependent costs through adaptive differential evolution.The approach is designed for cases where misclassification costs are unknown in practice.
  • Training process: Training begins by randomly initializing a population of misclassification costs and training a DBN with the training dataset.The candidate costs are then applied to the DBN outputs before performance evaluation.
  • Training process: Candidate costs are evaluated using cost-sensitive hypothesized predictions on the training data, and suitable costs are selected to generate the ECS-DBN.The supplied table identifies this sequence as the ECS-DBN training process.

A. Chromosome Encoding

ECS-DBN encodes misclassification costs directly as numerical chromosome variables and evolves them through differential-evolution operations. Candidate fitness is based on the G-mean of a DBN trained on the training data, after which the best chromosome supplies the final costs.

  • Chromosome Encoding: Each chromosome directly represents misclassification costs for the different classes as numerical values.The best evolved chromosome is used as the misclassification-cost vector for ECS-DBN.
  • Population and fitness: Initial candidates are sampled within the chromosome solution-space range and evaluated through corresponding trained DBNs.The fitness vector is obtained for each candidate from its trained DBN on the training set.
  • Population and fitness: The fitness function for each candidate is the G-mean of the training dataset.This evaluation links evolutionary cost optimization to the paper’s imbalanced-classification performance measure.
  • Final model: After evolution, the best individual supplies the misclassification cost used to create ECS-DBN and evaluate it on the test dataset.Adaptive parameter updates use successful mutation factors and crossover probabilities during evolution.

B. Population Initialization

Population initialization samples candidate misclassification costs uniformly within the feasible solution space. The resulting population forms the evolving unit, with each iteration treated as a generation.

  • Population Initialization: The initial population is generated by uniformly sampling each variable within its specified feasible range.Each individual is initialized using c_i^0 = c_min + rand(0, 1)·(c_max − c_min).
  • Population Initialization: The population holds possible misclassification costs and forms the unit of evolution.These candidate cost assignments are subsequently evolved through repeated generations.
  • Population Initialization: Each iteration of the misclassification-cost evolution process is called a generation.The population is repeatedly processed as the evolutionary search proceeds.

C. Adaptive DE Operators

Adaptive differential evolution evolves candidate solutions through mutation, crossover, and selection to optimize misclassification costs for ECS-DBN. The evaluation uses training-set G-mean, and the resulting best individual supplies the costs for ECS-DBN.

  • C. Adaptive DE Operators: Adaptive differential evolution updates the population through mutation, binomial crossover, and selection across generations.The algorithm uses these three evolutionary operations sequentially.
  • C. Adaptive DE Operators: Each chromosome represents class-specific misclassification costs, and the best evolved chromosome becomes the ECS-DBN costs.This encoding avoids requiring prior knowledge of the appropriate costs.
  • C. Adaptive DE Operators: Training-set G-mean is the objective function used to evaluate candidate misclassification costs.Each chromosome is introduced into an individual DBN as its misclassification costs during fitness evaluation.
  • C. Adaptive DE Operators: The optimization terminates at a maximum generation limit or when the best fitness remains unchanged for 30 generations.The unchanged-fitness condition is the stated convergence criterion.
  • C. Adaptive DE Operators: The evaluation covers 58 KEEL binary-class imbalanced benchmark datasets with reported imbalance ratios, attributes, and train/test counts.The datasets are selected from the KEEL repository and described as having high imbalance ratios.

C. Implementation Details

The implementation evaluates seven DBN-based algorithms on 58 KEEL benchmark datasets using accuracy and G-mean. Across these datasets, ECS-DBN achieves the strongest reported overall performance, particularly for G-mean, while resampling methods are compared under repeated cross-validation.

  • C. Implementation Details: All simulation results use 5-fold cross-validation over 10 trials on the 58 benchmark datasets.The experiments run on an Intel Core i5 machine with 16 GB RAM and an NVIDIA GeForce GTX 980.
  • C. Implementation Details: Accuracy and G-mean are the evaluation metrics, with G-mean accounting for performance on both minority and majority classes.Higher G-mean indicates better performance across both classes, and the metric is less sensitive to class distributions.
  • C. Implementation Details: The study compares ECS-DBN, DBN, ADASYN-DBN, SMOTE-DBN, two borderline-SMOTE variants, and SMOTE-SVM-DBN.The resampling variants generate synthetic minority data for comparison with ECS-DBN.
  • C. Implementation Details: ECS-DBN outperforms on 52 of 58 datasets for G-mean and excels on 34 of 58 datasets for accuracy.The authors attribute the G-mean result to evolutionary optimization using maximized G-mean as the objective.
  • C. Implementation Details: ECS-DBN achieves higher average values and lower variance than competing methods for both G-mean and accuracy across 58 benchmark datasets.Figure 3 summarizes the overall comparison across the seven algorithms.

F. Computational Time Analysis

ECS-DBN is evaluated against competing imbalance-learning methods using statistical tests and average ranks. The method generally outperforms competitors, ranks first across most benchmark datasets, and incurs higher computational cost from evolutionary optimization.

  • F. Computational Time Analysis: ECS-DBN has higher computational cost than the compared resampling methods, mainly because of the evolutionary algorithm.The resampling methods are somewhat faster on the small KEEL datasets.
  • F. Computational Time Analysis: Runtime is closely related to DBN network complexity: larger and deeper networks require more computation.Table III reports average computational time using 5-fold cross-validation over 10 trials across 58 datasets.
  • F. Computational Time Analysis: Wilcoxon paired signed-rank tests at the 5% significance level compare ECS-DBN with other methods separately for each dataset.The analysis counts win-lose-draw outcomes for accuracy and G-mean comparisons.
  • F. Computational Time Analysis: Holm post-hoc tests use ECS-DBN as the control algorithm for multiple comparisons of accuracy and G-mean.The reported p-values indicate statistically significant superiority over competing methods.
  • F. Computational Time Analysis: ECS-DBN ranks first across most benchmark datasets and outranks the other algorithms in both G-mean and accuracy.Average rank is the mean rank of each method across all datasets.
  • F. Computational Time Analysis: The authors conclude that ECS-DBN significantly outperforms competing methods, supporting cost-sensitive learning and the effectiveness of optimization.The conclusion combines Wilcoxon results, Holm tests, and average-rank evidence.

VI. EVALUATION ON A REAL-WORLD DATASET

The gun drilling study evaluates DBN and imbalance-learning methods on sensor-based tool-condition data using multiple classification metrics. The experiments capture machining signals and tool wear under controlled drilling conditions.

  • Experimental setup: The dataset uses accelerometers, a dynamometer, and a microscope to measure vibration, force, torque, and tool wear.
  • Evaluation metrics: The evaluation compares accuracy, G-mean, AUC, precision, and F1-score on the imbalanced gun drilling dataset.
  • Experimental setup: The gun drilling experiments specify tool diameter, spindle speed, feed rate, machining time, and six tool geometries across 20 drilling inserts.
  • Evaluation results: DBN achieves higher average accuracy, G-Mean, AUC, precision, and F1-score than the compared machine-learning methods.
  • Evaluation metrics: F1-score evaluates the faulty minority class, while G-mean and F1-score capture tradeoffs and AUC evaluates performance across both classes.

D. Experiment Results

The experiments compare ECS-DBN with conventional DBN, cost-sensitive alternatives, and resampling methods. ECS-DBN improves key imbalance-sensitive metrics and uses optimized misclassification costs to improve minority-class performance.

  • Comparative evaluation: ECS-DBN is compared with DBN, cost-sensitive methods, and resampling-based DBN variants on benchmark and real-world experiments.
  • Comparative evaluation: ECS-DBN obtains better performance than other imbalance-learning methods in G-mean, AUC, and precision.
  • Baseline comparison: DBN outperforms nine conventional machine-learning algorithms across the reported evaluation metrics on the gun drilling dataset.
  • Comparative evaluation: ECS-DBN outperforms WELM and ECO-ensemble and performs better than resampling methods on G-mean and precision.
  • Cost optimization: Optimized misclassification costs improve ECS-DBN performance over grid search and provide comparable accuracy and F1-score.
  • Class-specific effects: ECS-DBN improves minority-class accuracy while trading off performance between majority and minority classes.

E. Computational Cost

The computational-cost evaluation compares ECS-DBN with DBN and resampling-based methods on the gun drilling dataset. ECS-DBN is reported as more efficient for the large dataset.

  • Computational comparison: ECS-DBN consumes less average computational time than DBN and the evaluated resampling-based methods.
  • Computational comparison: The large number of gun drilling samples increases resampling complexity, while evolutionary cost estimation requires very little time relative to DBN training.
  • Computational comparison: Average computational time is compared across seven algorithms using 5-fold cross-validation over 10 trials.

VII. CONCLUSION

ECS-DBN combines cost-sensitive learning with adaptive differential evolution to address unknown misclassification costs in imbalanced classification. It outperformed DBN and resampling-based alternatives across benchmark datasets, while remaining computationally efficient on large-scale data.

  • ECS-DBN significantly outperformed competing techniques on 58 benchmark datasets and a real-world dataset.The approach applies cost-sensitive learning to DBN and uses adaptive differential evolution to find unknown misclassification costs.
  • ECS-DBN was computationally more efficient than some popular resampling methods on large-scale datasets and can be implemented in multiclass scenarios.
  • Future extensions include applying cost-sensitive methods to high-dimensional and dynamic data, online imbalanced classification, and other deep learning models.
  • ECS-DBN outperformed DBN and six resampling-based methods in test accuracy on 34 of 58 KEEL benchmark datasets.
  • ECS-DBN outperformed DBN and six resampling-based methods in test G-mean on 51 of 58 KEEL benchmark datasets.
Loading 1804.10801v2…