Source-linked AI summary
Review of Swarm Intelligence-based Feature Selection Methods
Mehrdad Rostami, Kamal Berahmand, Saman Forouzandeh
TL;DR
High-dimensional datasets make feature selection important for reducing irrelevant or redundant features and computational complexity. This paper categorizes and reviews swarm-intelligence-based methods, finding that PSO generally performs best among wrapper methods while ACO performs best among evaluated filter methods and balances exploration and exploitation most effectively.
Problem
High-dimensional datasets often contain irrelevant or redundant features, increasing prediction-model complexity and reducing performance.
Method
The paper categorizes feature-selection methods, reviews recent swarm-intelligence approaches, and compares wrapper- and filter-based methods across classifiers and datasets.
Results
PSO generally performs best among wrapper methods, whereas ACO achieves the highest average accuracy among evaluated filter methods and best balances exploration and exploitation.
Takeaways & Limitations
Swarm-intelligence feature-selection methods can substantially reduce dimensionality, but their relative performance depends on the feature-selection approach and classifier.
Takeaways & Limitations
Wrapper methods can be expensive to run and may break down on high-dimensional medical datasets because they use learning algorithms to evaluate feature subsets.
Abstract
from arXiv · showhide
In the past decades, the rapid growth of computer and database technologies has led to the rapid growth of large-scale datasets. On the other hand, data mining applications with high dimensional datasets that require high speed and accuracy are rapidly increasing. An important issue with these applications is the curse of dimensionality, where the number of features is much higher than the number of patterns. One of the dimensionality reduction approaches is feature selection that can increase the accuracy of the data mining task and reduce its computational complexity. The feature selection method aims at selecting a subset of features with the lowest inner similarity and highest relevancy to the target class. It reduces the dimensionality of the data by eliminating irrelevant, redundant, or noisy data. In this paper, a comparative analysis of different feature selection methods is presented, and a general categorization of these methods is performed. Moreover, in this paper, state-of-the-art swarm intelligence are studied, and the recent feature selection methods based on these algorithms are reviewed. Furthermore, the strengths and weaknesses of the different studied swarm intelligence-based feature selection methods are evaluated.
1. Introduction
Feature selection searches for relevant, non-redundant feature subsets to improve data-mining tasks, while swarm-intelligence-based methods have achieved acceptable performance but remain insufficiently reviewed. This paper compares and categorizes feature-selection methods, reviews recent SI-based approaches, and evaluates their strengths and weaknesses.
- Feature Selection: Feature selection searches candidate subsets, evaluates them until a stopping criterion, and validates the best subset on test data.The total search space includes 2^n possible subsets, where n is the number of features.
- Swarm Intelligence: Swarm-intelligence algorithms have been applied to feature selection with acceptable performance, but few studies comprehensively review SI-based methods.An earlier comparison focused only on well-known traditional swarm-intelligence algorithms and their feature-selection methods.
- Paper Scope: The paper categorizes feature-selection methods, reviews recent swarm-intelligence-based approaches, and evaluates their strengths and weaknesses.The paper also reports experimental results for different SI-based methods before presenting its conclusion.
2. Background
Feature selection addresses the curse of dimensionality by reducing irrelevant, redundant, or noisy features, with supervised methods generally offering more efficient and reliable performance than unsupervised methods. The background distinguishes filter, wrapper, hybrid, embedded, and graph-based approaches, highlighting trade-offs between efficiency, accuracy, computational cost, and feature relationships.
- Background: High dimensionality can increase classification parameters and significantly reduce classifier performance, motivating feature selection for dimensionality reduction.This problem is especially noted in medical applications with very high-dimensional datasets.
- Background: Supervised feature selection generally achieves better efficiency and more reliable performance than unsupervised selection because class labels guide feature evaluation.Unsupervised selection is more difficult because training data lack class tags.
- Filter methods: Filter methods evaluate feature relevance from inherent statistical and probabilistic data properties independently of machine learning algorithms.They are categorized into univariate and multivariate approaches according to how features are evaluated.
- Wrapper methods: Wrapper methods search for feature subsets using a classifier, but their potentially better selections are expensive and can fail on high-dimensional medical datasets.The classifier or learning model evaluates each generated subset during the search.
- Other feature selection models: Hybrid methods aim to balance filter efficiency with wrapper accuracy, while embedded methods incorporate feature selection directly into the learning algorithm.Graph-based methods additionally model similarity relationships and underlying manifold structure among features.
3. Swarm intelligence-based feature selection
Swarm intelligence-based feature selection addresses the computational difficulty of selecting useful feature subsets in high-dimensional data. The section distinguishes feature ranking from subset selection and situates swarm intelligence among meta-heuristic optimization approaches.
- Feature selection categories: Feature selection methods are categorized as feature ranking, which scores and removes weak features, or subset selection, which searches possible feature subsets.Ranking evaluates features individually under a criterion, whereas subset selection evaluates combinations of features.
- Feature subset selection: Feature subset selection is NP-hard, making exhaustive search impractical for medium-sized datasets despite guaranteeing an optimal subset.The computational cost of evaluating all possible subsets motivates searching for solutions that balance quality and complexity.
- Meta-heuristic optimization: Meta-heuristic feature selection methods use iterative optimization to search for good solutions while avoiding the complexity of evaluating every subset.These methods commonly initialize a population of random solutions and improve their optimality across iterations until a termination criterion is met.
- Meta-heuristic categories: Meta-heuristics are classified into evolutionary algorithms and swarm intelligence, with evolutionary algorithms using reproduction, mutation, recombination, and selection mechanisms.Candidate solutions are progressively modified during repeated generations or iterations.
3.1. PSO-based methods · 3.2. ACO-based methods
Sections 3.1 and 3.2 review feature-selection methods based on Particle Swarm Optimization and Ant Colony Optimization, including filter, wrapper, MRMR, similarity, and probabilistic approaches. The reviewed methods apply these swarm algorithms to dimensionality reduction and feature-subset optimization across classification, regression, detection, and unsupervised settings.
- 3.1. PSO-based methods: PSO, introduced by Kennedy and Eberhart in 1995, is an evolutionary swarm-intelligence algorithm inspired by the social behavior of birds and fish.The review notes that PSO has been used in many studies to solve feature-selection problems.
- 3.1. PSO-based methods: A PSO-based wrapper method combines a univariate filter model with wrapper-based feature selection.
- 3.1. PSO-based methods: PSO can optimize an MRMR-based single-objective function to reduce original acoustic features before neural-network defect detection.
- 3.2. ACO-based methods: ACO originated from the Ant System heuristic and was proposed as a multi-agent algorithm for optimization problems, initially including the traveling-salesman problem.
- 3.2. ACO-based methods: One ACO feature-selection formulation represents the problem as a clustered graph and evaluates generated subsets using a filter criterion without learning.
- 3.2. ACO-based methods: Unsupervised probabilistic feature selection with ACO iteratively searches for an optimal subset using similarity between features.
3.3. ABC-based methods · 3.4. DE-based methods
Sections 3.3 and 3.4 review feature-selection methods based on Artificial Bee Colony and Differential Evolution swarm-intelligence algorithms. They describe each algorithm’s search characteristics and representative applications to feature selection and clustering.
- 3.3. ABC-based methods: ABC simulates bees’ food-search behavior and combines local and random search in its early formulation.
- 3.3. ABC-based methods: ABC has been used in multiple studies to search for optimal feature subsets.
- 3.4. DE-based methods: A multi-objective DE feature-selection approach is proposed to improve clustering performance simultaneously.
3.5. GSA-based methods · 3.6. FA-based methods
Sections 3.5 and 3.6 review feature-selection methods based on the Gravitational Search Algorithm (GSA) and Firefly Algorithm (FA). GSA is inspired by Newtonian gravitation, whereas FA derives from optical communication between fireflies.
- 3.5. GSA-based methods: GSA is a physics-based swarm intelligence algorithm inspired by Newton’s law of universal gravitation.
- 3.5. GSA-based methods: GSA is described as a popular swarm intelligence technique widely employed in data management.
- 3.5. GSA-based methods: Recent research has proposed many feature-selection methods using GSA-based optimization.
- 3.6. FA-based methods: The Firefly Algorithm was introduced by Xin-She Yang in 2010 and was inspired by fireflies’ optical communication.
- 3.6. FA-based methods: FA exemplifies swarm intelligence in which low-performance agents cooperate to achieve high-performance results.
- 3.6. FA-based methods: A novel FA-based feature-selection method called return-cost-based binary FFA was developed by Zhang, Song, and colleagues.
3.7. BA-based methods
The Bat Algorithm (BA) is a swarm-intelligence method inspired by bats’ echolocation behavior. A feature-selection approach combines robust mRMR filtering with an improved BA search strategy to select the final feature subset.
- 3.7. BA-based methods: BA is a swarm-intelligence algorithm inspired by bats’ echolocation behavior.Bats locate prey by sending sound waves and receiving their reflections.
- 3.7. BA-based methods: Bats use emitted sound waves and returned reflections to find prey paths and locations.
- 3.7. BA-based methods: The feature-selection method uses robust mRMR to select relevant features and an improved BA algorithm to search for the final subset.
3.8. COA-based methods
This section introduces Cuckoo Optimization Algorithm (COA) as a swarm-intelligence algorithm inspired by cuckoo behavior, particularly egg-laying and reproduction. Like other evolutionary algorithms, COA begins with an initial population of cuckoos carrying eggs.
- 3.8. COA-based methods: COA is a swarm-intelligence optimization algorithm inspired by the cuckoo’s egg-laying and reproduction behavior.Its formation is based on the cuckoo’s distinctive lifestyle and reproductive habits.
- 3.8. COA-based methods: COA begins with an initial population of cuckoos, each associated with a number of eggs.
3.9. GWO-based methods
GWO is a bio-inspired meta-heuristic that models gray-wolf leadership and hunting mechanisms, and it has recently been applied to feature selection in data mining.
- 3.9. GWO-based methods: GWO simulates gray wolves’ leadership hierarchy and hunting mechanism as a bio-inspired optimization approach.The algorithm is inspired by the hunting process of gray-wolf packs in nature.
- 3.9. GWO-based methods: Recent studies have used GWO-based methods, including a multi-objective GWO approach, for feature selection in data mining.
3.10. WOA-based methods
WOA-based feature-selection methods adapt a whale-inspired swarm optimizer to explore and exploit feature subsets. Reviewed approaches include simulated-annealing and evolutionary operators, alongside a frequency-based filter method.
- 3.10. WOA-based methods: WOA models humpback-whale hunting through prey-search, encircling, and bubble-net foraging operators.The algorithm is inspired by humpback whales’ hunting behavior.
- 3.10. WOA-based methods: A hybrid WOA–simulated annealing method uses simulated annealing to enhance exploitation in promising regions identified by WOA.The hybrid combines WOA-based localization with simulated annealing’s focused search.
- 3.10. WOA-based methods: A WOA wrapper feature-selection approach enhances exploration and exploitation using tournament and roulette-wheel selection, crossover, and mutation.These strategies and operators are used to improve the swarm-intelligence search process.
3.11. SSA-based methods
SSA is a swarm-intelligence algorithm inspired by salps’ ocean swarming and foraging behavior. Reviewed SSA-based feature-selection studies include binary, hybrid, single-objective, and multi-objective wrapper approaches across numerical, medical, and other datasets.
- 3.11. SSA-based methods: SSA is inspired by the swarming behavior of salps while moving and foraging in oceans.The algorithm was proposed as a bio-inspired optimization method based on swarm intelligence.
- 3.11. SSA-based methods: A binary SSA with a crossover scheme was proposed to improve feature-selection accuracy, alongside a hybrid SSA-based optimization method.These methods were reported in studies by Faris, Mafarja et al. (2018) and Ibrahim, Ewees et al. (2019), respectively.
- 3.11. SSA-based methods: SSA-based feature-selection research spans six reviewed studies, predominantly using single-objective wrappers, with one multi-objective wrapper study.The reviewed applications cover numerical and medical datasets.
4. Experimental results
The experiments compare swarm-intelligence feature-selection methods using classification accuracy and selected-feature counts across multiple datasets and classifiers. Results identify classifier-dependent leaders and substantial dimensionality reduction.
- Experimental setup: Each method was run ten times after dataset normalization, with each dataset randomly split into 66% training and 34% test data.Max-Min normalization mapped values to [0 1], and missing values were replaced by averages of available data.
- Experimental setup: Methods were evaluated over SpamBase, Sonar, Arrhythmia, Madelon, Isolet, and Colon using SVM, Naïve Bayes, and AdaBoost classifiers.The datasets were taken from the UCI repository and selected for diverse characteristics.
- Wrapper-based methods: PSO-based methods ranked highest with SVM and AdaBoost, whereas COA-based ranked highest with Naive Bayes; PSO-based SVM accuracy reached 89.35%.Figures 1–3 report average accuracy across all datasets and classifiers.
- Wrapper-based methods: PSO-based selection achieved the strongest dimensionality reduction on Arrhythmia, Madelon, Isolet, and Colon, selecting 7.21%, 14.87%, 22.95%, and 0.58%, respectively.Table 5 reports selected-feature counts for eleven wrapper methods across datasets.
- Filter-based methods: ACO-based methods ranked highest across all three classifiers in the filter-based comparison, reaching 79.52% average accuracy with SVM.The SVM result led PSO-based methods by a 0.70 percentage-point margin.
5. Conclusions
High-dimensional datasets have grown substantially, while irrelevant or redundant features can reduce prediction-model performance. Feature selection is therefore essential for high-dimensional machine-learning datasets, including medical data.
- High-dimensional datasets have grown significantly alongside advances in data collection and storage technologies.
- Irrelevant or redundant features in high-dimensional datasets can reduce prediction-model performance.
- Feature selection plays an essential role in machine learning and reduces the size of medical datasets.