PaperlayerView exact revision on arXiv

Source-linked AI summary

Cost Sensitive Learning of Deep Feature Representations from Imbalanced Data

Salman H. Khan, Munawar Hayat, Mohammed Bennamoun, Ferdous Sohel, Roberto Togneri

arXiv:1508.03422v3cs.CV

TL;DR

Class imbalance makes minority classes difficult to learn and can bias classification toward majority classes. The paper introduces a cost-sensitive deep CNN that jointly learns class-dependent costs, feature representations, and classifier parameters without changing the original data distribution. Across six image-classification datasets, the modified cost functions outperform baselines and improve performance for both majority and minority classes.

  • Problem

    Class imbalance underrepresents minority classes, making it difficult to learn appropriate discriminating boundaries and correctly classify infrequent but important examples.

  • Method

    A cost-sensitive deep CNN automatically learns class-dependent costs jointly with robust feature representations and classifier parameters during training.

  • Results

    Across six classification datasets, the cost-sensitive CNN outperforms the baseline model, with average class-accuracy gains of 9.5% and 11.8% in two experiments.

  • Takeaways & Limitations

    The modified cost functions perform well on both majority and minority classes while preserving the original data distribution during training.

Abstract

from arXiv · show

Class imbalance is a common problem in the case of real-world object detection and classification tasks. Data of some classes is abundant making them an over-represented majority, and data of other classes is scarce, making them an under-represented minority. This imbalance makes it challenging for a classifier to appropriately learn the discriminating boundaries of the majority and minority classes. In this work, we propose a cost sensitive deep neural network which can automatically learn robust feature representations for both the majority and minority classes. During training, our learning procedure jointly optimizes the class dependent costs and the neural network parameters. The proposed approach is applicable to both binary and multi-class problems without any modification. Moreover, as opposed to data level approaches, we do not alter the original data distribution which results in a lower computational cost during the training process. We report the results of our experiments on six major image classification datasets and show that the proposed approach significantly outperforms the baseline algorithms. Comparisons with popular data sampling techniques and cost sensitive classifiers demonstrate the superior performance of our proposed method.

I. INTRODUCTION

The paper addresses class imbalance, where scarce minority classes are underrepresented and inadequately learned. It proposes jointly learning cost-sensitive CNN representations and classifier parameters, avoiding data redistribution while outperforming baseline and state-of-the-art approaches.

  • Motivation: Long-tailed data distributions leave minority classes underrepresented and bias classifiers toward majority classes.As a result, minority-class characteristics are not adequately learned.
  • Motivation: Correctly classifying infrequent but important examples, such as melanoma lesions or dangerous activities, motivates improving overall accuracy without sacrificing class precision.
  • Proposed approach: The paper jointly learns robust feature representations and classifier parameters under a cost-sensitive setting.Class-dependent costs are incorporated directly into CNN training rather than by changing the training data distribution.
  • Proposed approach: Class-dependent costs are automatically set from data statistics instead of relying on a handcrafted cost matrix.The costs use statistics such as data distribution and separability measures during learning.
  • Contributions: The method introduces cost-sensitive versions of three common CNN loss functions and alternately optimizes network parameters and class-sensitive costs for binary and multi-class classification.The paper also analyzes the resulting back-propagated gradients and reports no significant effect on training or testing time.
  • Results: The proposed approach is reported to outperform baseline procedures and state-of-the-art approaches.

B. Our Proposed Cost Matrix

The proposed cost matrix modifies CNN outputs before softmax and loss computation, perturbing classifier confidences to emphasize infrequent and difficult-to-separate classes. Its constrained positive values are intended to support smooth parameter updates during training.

  • Cost-matrix operation: The cost matrix modifies the CNN output before the softmax and loss layers, and the resulting activations are squashed to [0, 1].
  • Cost-matrix operation: Score-level costs perturb classifier confidences to encourage correct classification of infrequent classes.The perturbation also gives more importance to less frequent and difficult-to-separate classes.
  • Cost-matrix operation: The modified output is computed as y(i) = F(ξ_p, o(i)), where F depends on the loss layer.For cost-sensitive MSE, F is defined as sigmoid(ξ_p ◦ o(i)).
  • Matrix properties: For a cost-insensitive loss, ξ is an all-ones matrix rather than the traditional 1−I matrix.
  • Matrix properties: All entries of ξ are positive and lie in (0, 1], while column offsets can produce an equally probable guess point.
  • Matrix properties: These matrix properties avoid excessively increasing CNN activations or reducing them to zero, enabling smooth training and parameter updates.

C. Cost-Sensitive Surrogate Losses

The paper introduces cost-sensitive versions of common surrogate losses that incorporate class-dependent penalties into CNN training. These modified losses retain classification-oriented properties while enabling joint learning of network parameters and costs.

  • The cost-sensitive error is defined as the mean loss over the training set.
  • Cost-Sensitive MSE: Cost-sensitive MSE minimizes squared error between predicted weighted outputs and ground-truth labels.
  • Cost-Sensitive SVM Hinge Loss: Cost-sensitive SVM hinge loss maximizes the margin between each pair of classes.
  • Cost-Sensitive CE Loss: Cost-sensitive CE uses class-dependent costs in the soft-max output before computing classification loss.
  • Classification Feasibility: The modified losses are shown to satisfy classification calibration and guess aversion, making them suitable for classification.

D. Optimal Parameters Learning

The method jointly learns CNN parameters and class-dependent costs through alternating optimization. Cost updates use validation error, class-to-class separability, classification errors, and training-class distributions.

  • The model jointly learns hypothesis parameters θ and class-dependent loss parameters ξ.
  • The algorithm alternates stochastic-gradient updates of θ with gradient-descent updates of ξ.
  • Class-to-class separability is computed from intra-class and inter-class nearest-neighbor distances in penultimate-layer feature space.
  • Cost parameters are updated using a validation-set objective incorporating classification errors, separability, and class-distribution statistics.
  • The validation error controls cost-parameter updates, including reducing the ξ learning rate when validation performance worsens.

E. Effect on Error Back-propagation

The modified loss functions alter the gradients used by CNN error back-propagation.

  • The section analyzes how modified loss functions affect gradient computation during CNN back-propagation.

1) Cost-Sensitive MSE:

For cost-sensitive MSE, the class-dependent weighting changes the output-layer gradients computed during supervised CNN training.

  • Cost-sensitive MSE minimizes mean squared error between predicted weighted outputs and ground-truth labels across the training set.
  • The loss modification changes the gradient computed during back-propagation at each output-layer neuron.

2) Cost-Sensitive SVM Hinge Loss:

The cost-sensitive SVM hinge-loss section derives neuron-level directional derivatives and combines them with softmax derivatives to characterize back-propagation.

  • The directional derivative of the SVM hinge loss is computed at each neuron.
  • The softmax output derivative with respect to the penultimate-layer output is combined with the hinge-loss derivative to represent the loss derivative.
  • The derivation uses an indicator function to express the relevant derivative conditions.

3) Cost-Sensitive CE loss:

The cost-sensitive cross-entropy loss incorporates class-dependent costs through the softmax outputs while preserving the standard back-propagation gradient formulas. Its cost-modified surface shifts classification boundaries according to the relative costs.

  • The cost-sensitive softmax log loss introduces class-dependent costs while retaining the cross-entropy formulation.
  • The introduction of the class-imbalance cost does not affect gradient computation during back-propagation.
  • The proof differentiates the softmax neuron with respect to its input and treats the cases m = n and m ≠ n separately.
  • The loss surface and classification boundaries are reshaped by score-level costs, favoring classes with relatively lower cost.
  • The loss is differentiated using the target distribution constraint that the output-class probabilities sum to one.
  • The costs affect the softmax output y_m, while the gradient formulas remain unchanged from the cost-insensitive cross-entropy case.
  • Experiments report only the cost-sensitive cross-entropy loss because it is reported to outperform the other two losses and avoids the MSE loss’s learning slowdown.

IV. EXPERIMENTS AND RESULTS

The experiments evaluate classification under varied imbalance protocols across six image datasets using accuracy, class recall, G-mean, and F-measure. They also note that AUC is omitted because the neural networks produce a single classification score.

  • The datasets span naturally imbalanced, protocol-balanced, and equally represented training and testing distributions.
  • The experiments report overall classification accuracy to compare baseline, state-of-the-art, balanced-data, and unbalanced-data approaches.
  • Class recall is reported through confusion matrices, while G-mean and F-measure are evaluated across all six datasets.
  • AUC measurements are omitted because the neural networks provide a single classification score, making ROC curves unavailable.
  • Melanoma detection uses the Edinburgh Dermofit Image Library, containing 1,300 lesion images across 10 lesion types with three-fold cross-validation.
  • Coral classification uses 2,055 images with roughly 400,000 labeled points across nine classes and varying class representation.
  • Caltech-101 experiments use both standard balanced protocols and 60%/40% and 30%/70% train/test splits.
  • MIT-67 is evaluated under its standard uniform split and complete-dataset imbalanced 60%/40% and 30%/70% train/test splits.

3) Balanced Datasets-Balanced Protocols:

The study evaluates a cost-sensitive CNN across balanced and deliberately imbalanced image-classification protocols, using six datasets and comparisons with baseline and alternative approaches. The method matches the baseline on balanced distributions but improves as imbalance increases, with especially strong gains for minority-class accuracy.

  • Datasets and protocols: Experiments cover MNIST, CIFAR-100, DIL, MLC, Caltech-101, and MIT-67 under standard and deliberately imbalanced training splits.For MNIST and CIFAR-100, minority-class representations are reduced to 25% or 10%; DIL and MLC use imbalanced protocols, while Caltech-101 and MIT-67 include standard and original distributions.
  • Model: The CoSen CNN uses an 18-weight-layer deep CNN with a cost-sensitive loss layer, while the baseline has the same architecture with a conventional cross-entropy loss.The first 16 layers are initialized from a pretrained model, and the final two fully connected layers receive random initialization before full-network training.
  • Imbalanced datasets: The method improves average class accuracy by 9.5% and 11.8% on the reported DIL and MLC experiments, respectively.The gains are especially pronounced for least frequent classes, including Turf, Macro, Monti, AK, and SCC.
  • Balanced datasets: On balanced training distributions, the proposed approach performs equally to the baseline, whereas increasing imbalance produces significant improvements over the baseline CNN.This pattern is reported for the balanced datasets under standard and artificially imbalanced splits.
  • Comparisons and efficiency: The approach significantly outperforms data-sampling and cost-sensitive classifier alternatives on four classification datasets with highly imbalanced training sets.Adaptive costs also outperform fixed costs based on class representation, separability, or classification errors, while adding only modest training overhead and no test-time overhead.

V. CONCLUSION

The paper proposes a cost-sensitive deep CNN that automatically learns class-dependent costs and network parameters for imbalanced classification. Across six datasets, the modified losses perform well for both majority and minority classes.

  • Conclusion: The proposed cost-sensitive deep CNN automatically sets class-dependent costs from training-set statistics to address class imbalance.The costs incorporate data distribution, class separability, and classification errors through alternating optimization.
  • Conclusion: Three cost-sensitive loss functions are introduced, with the cost-sensitive cross-entropy loss shown to be c-calibrated and guess-aversive.
  • Conclusion: Experiments on six popular classification datasets show that the modified cost functions perform well on majority and minority classes.

APPENDIX A PROOFS REGARDING COST MATRIX ξ′

The appendix establishes properties of the cost matrix used in the analysis, including invariance to column offsets and constraints on admissible costs. These results characterize when cost transformations preserve risk or lead to random guessing.

  • Cost-matrix invariance: Adding a constant to every column of the cost matrix ξ′ leaves the associated classification risk unchanged.
  • Cost constraints: For a correct prediction, the true-class cost should be lower than the mean cost of all misclassifications.
  • Cost-insensitive case: The cost-insensitive matrix ξ is an all-ones matrix, so all decisions have uniform cost and CNN activations remain unchanged.
  • Cost constraints: All elements of the cost matrix ξ must lie in the interval (0, 1].
  • Random-guess boundary: Setting ξ to zero makes CNN outputs zero and causes the classifier to make a random guess.
Loading 1508.03422v3…