Source-linked AI summary
Survey on Feature Selection
Tarek Amr Abdallah, Beatriz de La Iglesia
TL;DR
Feature selection addresses the computational and modeling challenges posed by excessive, irrelevant, and redundant features in data mining. This review synthesizes selection processes and evaluation strategies in relation to learning algorithms, concluding that dataset and learner characteristics determine the relevant trade-offs. A key scope boundary is that an individually irrelevant feature can still contribute to an optimal subset.
Problem
Excessive, irrelevant, and redundant features burden learning algorithms and affect efficiency, accuracy, comprehensibility, bias–variance balance, and high-dimensional data analysis.
Method
The review organizes feature selection into search, evaluation, and stopping processes, and relates filter, wrapper, subset, and search approaches to learning algorithms.
Results
Dataset nature, classifier robustness, and learning-problem nature dictate choices among trade-offs involving filters, wrappers, individual or subset selection, and search algorithms.
Takeaways & Limitations
Feature-selection choices should be matched to the dataset, classifier, and learning problem rather than treated as universally optimal.
Takeaways & Limitations
An individually irrelevant feature can still form part of an optimal subset.
Abstract
from arXiv · showhide
Feature selection plays an important role in the data mining process. It is needed to deal with the excessive number of features, which can become a computational burden on the learning algorithms. It is also necessary, even when computational resources are not scarce, since it improves the accuracy of the machine learning tasks, as we will see in the upcoming sections. In this review, we discuss the different feature selection approaches, and the relation between them and the various machine learning algorithms.
1 Introduction
Feature selection supports both predictive and descriptive learning by removing irrelevant or redundant features. It improves efficiency, accuracy, and model comprehensibility while addressing bias–variance and high-dimensionality problems.
- Classification predicts discrete class labels from input features, whereas clustering categorizes data without supervised training.
- Removing irrelevant and redundant features makes data mining more efficient, improves accuracy, and simplifies the inferred model.
- Feature selection seeks a bias–variance trade-off by balancing variance reduction from fewer features against bias reduction from estimating more features.
- High-dimensional data makes proximity measures more uniform, making distance and density harder to obtain for unsupervised learning.
2 The selection process
Feature selection searches, evaluates, and stops over candidate feature subsets. Search strategies range from greedy and backtracking methods to genetic algorithms and staged individual-then-subset selection.
- The selection process consists of generating a feature subset, evaluating its utility, and deciding whether to stop or continue.
- Because an N-dimensional feature space has 2^N possible subsets, search may stop at a feature, iteration, or evaluation threshold instead of exploring all subsets.
- Forward selection adds features from an empty set, while backward elimination removes features from the full set; neither reverses its individual decisions.
- Best-first search generates all possible moves and can backtrack, but a stoppage criterion limits exploration of the search space.
- Genetic algorithms represent subsets as binary strings and use mutation and crossover to generate new candidate subsets.
- Staged selection first removes irrelevant features individually, then removes redundant features through subset selection to reduce the later search space.
- An irrelevant feature can still contribute to an optimal subset, showing that individual relevance does not fully determine subset usefulness.
3 Filters and Wrappers
Filters evaluate features independently of learning algorithms, whereas wrappers use the learner itself. Filters are faster, while wrappers can be more accurate because they account for learner-specific performance.
- Filters evaluate individual features or subsets independently of the learning algorithm, while wrappers evaluate subsets with the learning algorithm.
- Common filter criteria include mutual information, chi-square tests, and Pearson correlation coefficients.
- Mutual information ranks individual features by their information with the target class and selects the top n features.
- Mutual information can score rare features higher than common ones when their conditional probabilities for a class are equal.
- Information Gain and Mutual Information are distinguished by some papers, but the cited formula is also identified as Information Gain.
- mRMR addresses multivariate density limitations by combining inter-feature and feature-to-class mutual information.
- Laplacian Score selects features using neighborhood similarity without target labels, and becomes Fisher Criterion Score when labels are available.
- Wrappers commonly use cross-validation accuracy to evaluate subsets, but their effectiveness requires greater computational cost than filters.
4 FS and Learning Algorithms
Feature selection interacts with learning-algorithm properties, including sensitivity to irrelevant or redundant features and implicit selection during training. Search methods and feature-selection methods can perform differently across algorithms, datasets, sample sizes, and data characteristics.
- Instance-based algorithms are vulnerable to irrelevant features, whereas Naive Bayes is more robust to them but vulnerable to redundant features.
- Decision trees can select relevant features implicitly during training through embedded feature-selection methods, but may still need earlier feature selection.
- Different search and feature-selection algorithms perform differently across learning algorithms, datasets, sample sizes, and data characteristics.
5 Conclusion
Feature-selection trade-offs depend on the dataset, classifier robustness, and learning problem rather than having one universally best choice. Efforts also aim to extend filter methods to subset selection and unsupervised learning.
- Wrappers are generally more accurate than filters, whereas filters are more computationally efficient.
- Dataset nature, classifier robustness, and learning-problem nature determine which feature-selection trade-offs are appropriate.
- Research efforts seek to make filter methods suitable for subset selection and unsupervised learning scenarios.