PaperlayerView exact revision on arXiv

Source-linked AI summary

Evolutionary Cost-sensitive Extreme Learning Machine

Lei Zhang, David Zhang

arXiv:1505.04373v2cs.CV

TL;DR

Conventional ELM assumes equal costs for different misclassifications, while realistic cost-sensitive tasks may require costs that users cannot readily specify. ECSELM uses evolutionary search to optimize the cost matrix automatically. In reported classifier comparisons, it statistically outperforms other methods at α=5%.

  • Problem

    ELM assumes equal misclassification losses, while cost-sensitive learning needs a cost matrix that users may not know or be able to define precisely.

    Source: §I · p. 2

  • Method

    ECSELM integrates misclassification costs into ELM and uses evolutionary backtracking search to optimize the cost matrix automatically.

    Source: §I · p. 2, B. Evolutionary CSELM · p. 4, C. Optimization · p. 4

  • Results

    At α=5%, statistical tests report that ECSELM outperforms other methods in the evaluated classifier comparisons.

    Source: B. Statistical Significance · p. 11

  • Takeaways & Limitations

    Experiments span human beauty, face recognition, face verification, and E-NOSE, applying the framework across varied cost-sensitive scenarios.

    Source: §X · p. 12

    The authors identify improving the evolutionary algorithm through more appropriate population generation as future work.

    Source: §X · p. 12

Abstract

from arXiv · show

Conventional extreme learning machines solve a Moore-Penrose generalized inverse of hidden layer activated matrix and analytically determine the output weights to achieve generalized performance, by assuming the same loss from different types of misclassification. The assumption may not hold in cost-sensitive recognition tasks, such as face recognition based access control system, where misclassifying a stranger as a family member may result in more serious disaster than misclassifying a family member as a stranger. Though recent cost-sensitive learning can reduce the total loss with a given cost matrix that quantifies how severe one type of mistake against another, in many realistic cases the cost matrix is unknown to users. Motivated by these concerns, this paper proposes an evolutionary cost-sensitive extreme learning machine (ECSELM), with the following merits: 1) to our best knowledge, it is the first proposal of ELM in evolutionary cost-sensitive classification scenario; 2) it well addresses the open issue of how to define the cost matrix in cost-sensitive learning tasks; 3) an evolutionary backtracking search algorithm is induced for adaptive cost matrix optimization. Experiments in a variety of cost-sensitive tasks well demonstrate the effectiveness of the proposed approaches, with about 5%~10% improvements.

I. INTRODUCTION

ELM analytically learns output weights but conventionally treats all misclassifications as equally costly, while realistic recognition tasks can assign different losses to different errors. ECSELM addresses both cost-sensitive ELM learning and the difficulty of defining a cost matrix by automatically optimizing it.

A. SLFN and ELM

  • ELM randomly selects input weights and hidden biases, then analytically determines output weights using the Moore-Penrose generalized inverse.The approach seeks small training error and output-weight norm for generalization.Source: §I · p. 1, A. SLFN and ELM · p. 2

I. INTRODUCTION

  • Existing ELM variants address issues such as kernels, sparsity, adaptive hidden-node growth, random-weight optimization, or sample imbalance, while ECSELM targets misclassification costs.The cited differential-evolution ELM optimizes random input weights and biases, unlike ECSELM’s cost-matrix focus.Source: §I · p. 1
  • Equal-loss assumptions can be unsuitable for access control, where accepting a stranger as a family member is described as more serious than rejecting a family member.The paper frames this asymmetry as a motivation for cost-sensitive recognition.Source: §I · p. 2
  • Cost-sensitive learning can use a predefined matrix to quantify mistake severity, but users may know only which error is worse and struggle to specify its cost.Manually predefined cost intervals still require user input and can bring large computational cost.Source: §I · p. 2
  • ECSELM combines cost-sensitive ELM learning with automatic cost-matrix optimization, addressing the open problem of defining costs in realistic tasks.The authors describe it as the first evolutionary cost-sensitive ELM proposal.Source: §I · p. 2

B. Kernel ELM

The section reviews kernel and weighted ELM variants before describing cost-sensitive ELM, which incorporates misclassification costs into learning and prediction. Its cost information vector combines sample weighting and the cost matrix, and the resulting output weights and decisions depend on that vector.

B. Kernel ELM

  • Kernel ELM applies Mercer’s condition to ELM; the supplied passage does not detail its formulation or prediction rule.The accompanying passage notes a representation for the case N<L.Source: B. Kernel ELM · pp. 2–3

C. Weighted ELM

  • Weighted ELM embeds a class-associated constant weight matrix to address imbalanced samples, assigning minority-class samples larger weights and majority-class samples smaller weights.The passage identifies two schemes, W1ELM and W2ELM.Source: C. Weighted ELM · p. 3

A. Cost-sensitive Extreme Learning Machine

  • CSELM integrates a cost matrix that assigns different costs to different misclassification types, with zero diagonal costs for correct classifications.Its cost information vector combines the weighting matrix and misclassification-cost matrix.Source: A. Cost-sensitive Extreme Learning… · p. 3
  • CSELM’s cost information vector trades off sample imbalance and misclassification loss, and the output weights and final decision depend on it.Unlike a constant sample-count-based weight matrix, this vector depends on the unsolved cost information.Source: A. Cost-sensitive Extreme Learning… · p. 3

B. Evolutionary CSELM

ECSELM extends CSELM by optimizing cost information to minimize prediction loss, rather than relying on an empirically chosen cost matrix. It uses evolutionary search to obtain an optimized cost matrix and corresponding output weights.

B. Evolutionary CSELM

  • ECSELM’s procedure initializes weights and a cost matrix, computes hidden-layer outputs and output weights, optimizes the cost information, then recomputes output weights.The method introduces evolutionary search into CSELM for cost-matrix optimization.Source: B. Evolutionary CSELM · pp. 3–4
  • Empirically chosen cost matrices may lead to poor generalization, so ECSELM searches for a cost matrix that minimizes total prediction loss within bounds.The objective is expressed over training samples using the CSELM decision function and a classification or regression loss.Source: B. Evolutionary CSELM · p. 4
  • Because output weights depend on the cost information vector formed from the weighting and cost matrices, the optimization is transformed to search over that vector.The optimal output weights are obtained alongside the optimal cost information vector.Source: B. Evolutionary CSELM · p. 4

C. Optimization

The optimization uses backtracking search optimization (BSA) to learn the cost matrix under boundary constraints. Its population-based search cycles through initialization, historical-population selection, recombination, and selection of updated solutions.

C. Optimization

  • BSA learns the cost matrix using population-based stochastic search with selection, mutation, and crossover operators.The paper characterizes BSA as having a simple structure and being effective and fast.Source: C. Optimization · p. 4
  • The framework initializes and evaluates a population, updates historical and solution populations, applies boundary control, and tracks the global minimum and optimal solution.The algorithm repeats these operations until the specified iteration limit.Source: C. Optimization · pp. 4–5
  • BSA proceeds through initialization, Selection-I, recombination, and Selection-II to update its population and solutions.The described recombination stage updates the solution population.Source: C. Optimization · pp. 4–5

IV. HUMAN BEAUTY DATA ANALYSIS

The paper uses human beauty analysis to evaluate its proposed method because beauty assessment is recognized as a cost-sensitive learning task. It applies ECSELM to facial, dressing, and vocal attractiveness data from eastern and western women.

  • Human beauty analysis is used to evaluate ECSELM because it is recognized as a cost-sensitive learning task.Source: §IV · p. 5
  • The public M2B database contains facial, dressing, and vocal datasets of women from eastern and western cultural groups.Source: §IV · p. 5
  • The experiments apply ECSELM to facial, dressing, and vocal attractiveness assessment using the M2B database.Source: §IV · p. 5

A. M2B Database

M2B experiments evaluate ECSELM on facial, dressing, and vocal attractiveness data, using both categorical recognition and attractiveness-score prediction. Across recognition tasks and cross-culture score prediction, the reported results favor ECSELM.

A. M2B Database

  • The experiments use 300-dimensional facial and dressing features and 50-dimensional vocal features, reduced by PCA.Source: A. M2B Database · p. 5

B. Parameters Setting

  • The hidden-neuron count L ranges from 100 to 500 and penalty parameter C from 20 to 230; optimization uses population size and epochs of 100.Source: B. Parameters Setting · p. 5

C. Attractiveness Assessment: Beauty Recognition

V. FACE DATA ANALYSIS

The face experiments test ECSELM on AR recognition and LFW verification. On AR, ECSELM achieves the highest reported recognition rate among the methods compared in the paper.

V. FACE DATA ANALYSIS

  • The AR experiments use 100-person face recognition data and compare ECSELM with general, subspace, ELM-based, and specialized cost-sensitive methods.The authors state that face recognition is used to test method usefulness, rather than as the paper’s specific focus.Source: §V · p. 7, A. Experiment on AR Dataset · p. 7

A. Experiment on AR Dataset

B. Experiment on LFW Dataset

The LFW experiments evaluate face verification using paired-face similarity features and compare ECSELM with ELM, cost-sensitive subspace, and metric-learning methods. ECSELM is reported to outperform the compared methods, including state-of-the-art metric-learning methods on both descriptors.

B. Experiment on LFW Dataset

  • LFW verification uses five similarity metrics to represent each pair as a five-dimensional input for a binary classifier.Faces are represented by 300-dimensional vectors after PCA; reported view-2 accuracies are means across 10-fold cross-validation.Source: B. Experiment on LFW Dataset · p. 8
  • ECSELM performs best among the state-of-the-art metric-learning methods for both LFW descriptors.The comparison uses the LBP and SIFT descriptors.Source: Table IX · p. 9, B. Experiment on LFW Dataset · p. 9
  • ECSELM outperforms cost-sensitive subspace methods and conventional ELM methods on LFW verification.ELM-based methods generally outperform subspace methods, but show higher standard deviations.Source: B. Experiment on LFW Dataset · pp. 8–9
  • SubSML is the strongest metric-learning method on both descriptors, while CSML and LMLML have no reported SIFT results.The missing SIFT results were not reported in the cited studies.Source: B. Experiment on LFW Dataset · p. 9
  • The authors propose integrating group metrics as input features and learning a binary classifier for face verification.View 2 uses 10-fold cross-validation with 600 pairs per fold, split evenly between similar and dissimilar pairs.Source: B. Experiment on LFW Dataset · pp. 8–9

VI. E-NOSE DATA ANALYSIS

The E-NOSE experiments assess gas recognition across six classes using sensor-response features. ECSELM has the best reported recognition results, while the authors characterize the broader evidence as preliminary and call for more large-scale testing.

VI. E-NOSE DATA ANALYSIS

  • The six-gas E-NOSE dataset uses six-dimensional sensor-response features, with two thirds of each class selected for training.The classes include formaldehyde, benzene, toluene, carbon monoxide, ammonia, and nitrogen dioxide.Source: §VI · p. 9
  • ECSELM achieves 97.17% ARR and 98.15% TRR on E-NOSE, the best recognition performance among the compared ELM-based and subspace methods.The comparison with existing methods also reports CSELM and ECSELM as the best-performing methods.Source: §VI · p. 10
  • The authors describe the application evidence as preliminary and say more tests on large-scale databases are needed to assess the methods’ potential.They also note that evolutionary algorithms do not guarantee a global optimum.Source: §VI · p. 10

VII. PERFORMANCE EVALUATION OF CLASSIFIERS

Classifier performance is analyzed with ROC curves, AUC, and confusion matrices across LFW, M2B, and E-NOSE data. The reported LFW ROC/AUC comparison favors ECSELM.

A. ROC, AUC and Confusion Matrix Analysis

B. Statistical Significance

The authors report statistically significant pairwise superiority for ECSELM and examine its sensitivity to the trade-off coefficient and hidden-neuron count.

B. Statistical Significance

  • ECSELM statistically outperforms the other methods at α=5% in pairwise t-tests; the Kruskal-Wallis test also demonstrates statistical significance.The authors test nine methods on multiple datasets.Source: B. Statistical Significance · pp. 10–11

VIII. PARAMETER SENSITIVITY ANALYSIS

  • ECSELM and standard ELM are not sensitive to trade-off coefficient C; better performance is reported at C=25, 2^10, and 2^20 for AR, LFW, and E-NOSE, respectively.Fig. 11 compares performance across C values on AR, LFW, and E-NOSE.Source: §VIII · pp. 11–12, Figure 11 · p. 12
  • ECSELM's performance varies little with hidden-neuron count L, while ELM performance drops sharply on LFW and E-NOSE when L exceeds 200.The reported best L values for AR, LFW, and E-NOSE are 300, 100, and 200, respectively.Source: §VIII · p. 12, Figure 12 · p. 12

IX. COMPUTATIONAL COMPLEXITY AND TIME ANALYSIS

The section analyzes ECSELM's computational cost relative to ELM and subspace methods, then identifies future directions for ELM research.

IX. COMPUTATIONAL COMPLEXITY AND TIME ANALYSIS

  • ECSELM's matrix-computation work within Algorithm 2's search loop has complexity O(N·epochs·m·n·p).Algorithm 2's search complexity is O(N·epochs), and the hidden-layer output matrix can be precomputed.Source: §IX · p. 12
  • KELM and E-ELM require more computation than ELM and WELM, attributed to higher-dimensional kernel-matrix output weights and evolutionary search.Source: §IX · p. 12
  • CSMFA takes 6731.9s, the longest reported runtime, largely because it computes a locality graph by searching k nearest neighbors for each input vector.Source: §IX · p. 12
  • ECSELM is faster than cost-sensitive subspace methods except CSLDA, and ELMs are much more computationally efficient than subspace methods.Source: §IX · p. 12

X. CONCLUSION AND FUTURE WORK

  • The paper concludes that ECSELM provides an evolutionary cost-sensitive ELM framework for automatically determining task-specific cost matrices, with experiments across several application scenarios.Source: §X · p. 12
  • Future directions include improving evolutionary-algorithm population generation, exploring ELM's deep-learning capability, and investigating ensemble ELMs and low-discrepancy input weights.Source: §X · p. 12
Loading 1505.04373v2…