Multi-Objective Hyperparameter Tuning and Feature Selection using Filter Ensembles

1 Introduction

• Hyperparameter optimization is expensive because model performance generally lacks an analytic representation and must be estimated by test-set evaluation or cross-validation.The introduction notes only a few model-specific analytic methods [15] [29].Source: §1 · p. 1
• Sparser models can improve interpretability and reduce prediction costs, and may also improve predictive performance by regularizing against overfitting.A small performance drop may be acceptable in exchange for substantial sparsity gains.Source: §1 · p. 2
• Treating feature selection as multi-objective avoids specifying a performance–sparsity trade-off in advance through a single penalty.The aim is to maximize predictive performance while minimizing the number of selected features.Source: §1 · p. 2
• Jointly optimizing features and hyperparameters is motivated by their likely interdependence, although it combines an exponential feature-subset space with mixed numeric-categorical hyperparameters.The choice of included features may influence which hyperparameter configuration performs well, and vice versa.Source: §1 · p. 2
• The proposed model-agnostic approaches are hybrid filter-wrappers: they optimize model performance while using filter-score ensembles to guide selection.Filters supply guidance alongside wrapper-based performance optimization.Source: §1 · p. 2

2 Related Work and Contributions

• Bayesian optimization is a popular hyperparameter-tuning method, but Gaussian-process surrogates do not typically scale well to high dimensions and large datasets.Random forests are an alternative surrogate used by SMAC.Source: §2 · p. 2
• Some AutoML frameworks combine hyperparameter optimization with filter-based feature selection, but their stated objective is predictive performance rather than a performance–sparsity trade-off.Such frameworks may produce complex models and large heterogeneous ensembles that are difficult to interpret and deploy [30].Source: §2 · pp. 2–3
• Earlier NSGA-II work on simultaneous tuning and selection used an SVM wrapper, whereas the cited approach is limited to SVMs and lacks the specialized operators described here.The authors state that those operators were necessary for good performance in their approach.Source: §2 · p. 3
• The paper addresses the absence of a general model-agnostic method for jointly optimizing hyperparameters and feature sets for performance and sparsity using NSGA-II and extended Bayesian optimization.The authors benchmark the approaches against suitable baselines across tasks and learning algorithms.Source: §2 · p. 3
• The NSGA-II approach adds specialized sampling and mutation operators that use filter ensembles to guide optimization.Genetic algorithms use recombination and mutation, with non-dominated sorting supporting multi-objective search.Source: §2 · p. 3
• The Bayesian optimization approach adapts multi-objective BO to feature selection using a filter ensemble.The paper extends heterogeneous filter ensembles by optimizing a parameterized ranking aggregator.Source: §2 · p. 3

3 Problem Statement

• The setup assumes n i.i.d. samples from a joint distribution over feature vectors and outcomes; datasets are characterized by sample count n and feature count p.The outcome space may, for example, be {-1, 1} for binary classification.Source: §3 · p. 3
• A binary feature configuration s indicates which features are included, and its Hamming weight determines the model's sparseness.The learning algorithm is parameterized by s and a hyperparameter vector λ drawn from a possibly mixed bounded space Λ.Source: §3 · pp. 3–4
• The bi-objective problem treats estimated generalization error and feature cost as separate objectives.Generalization error is estimated from in-sample data using a resampling technique such as cross-validation.Source: §3 · p. 4
• Benchmarks assign equal feature costs, making the feature-cost measure the fraction of selected features, ranging from 0 to 1.The method can be extended to arbitrary per-feature costs, but the benchmarks use c_i = 1/p.Source: §3 · p. 4

4 Multi-Objective Hyperparameter Tuning and Feature Selection: Two Approaches

• Both the model-based and evolutionary approaches use feature filters to accelerate the joint search over hyperparameters and feature selection.The approaches adapt popular multi-objective optimization methods to this task.Source: §4 · p. 4

4.1 Feature Filters and Filter Ensembles

• The filter ensemble combines multiple filters by weighted average rank aggregation, extending the heterogeneous approach to a parameterized ranking aggregator.Individual filter scores are rank-transformed and scaled to equidistant values from 0 to 1 before aggregation.Source: §4.1 · p. 4
• The ensemble's weighting parameter can be optimized, making the filter a hybrid filter-wrapper approach.The weights sum to one.Source: §4.1 · p. 4

4.2 Bayesian Optimization Approach

• Parego scalarizes objectives with a newly sampled random weight vector for each point, while BO alternates surrogate fitting with infill selection balancing exploitation and exploration.Parego requires a single surrogate-model fit for each proposed point; other multi-objective BO methods can substitute for it.Source: §4.2 · pp. 4–5
• Hierarchical hyperparameter dependencies, such as kernel parameters active only for a chosen kernel, are not considered.This bounds the BO setup to search spaces without those conditional dependencies.Source: §4.2 · p. 4

4.2.1 Individual filter selection

• Individual filter selection adds filter-index and feature-fraction hyperparameters, including the top-ranked ⌈p · ffrac⌉ features from the chosen filter.The feature-configuration search is exponential in p, motivating use of filter scores to reduce its dimensionality.Source: §4.2 · p. 5, §4.2.1 · p. 6

4.2.2 Filter ensemble selection

• Filter-ensemble selection tunes filter weights w and feature fraction ffrac, selecting the top ⌈p · ffrac⌉ features by the weighted ensemble score EFj(w).Figure 1 represents individuals in different optimization-method variants; λ, w, and s are vectors.Source: Figure 1 · p. 5, §4.2.2 · p. 6

4.3 Evolutionary Approach

• NSGA-II favors candidates close to the Pareto front through nondominated sorting and iterates reproduction, crossover, mutation, and survival steps.The feature vector s receives specialized handling because its Hamming weight directly determines the selected-feature fraction objective and can occupy much of the search space.Source: §4.3 · p. 6

4.3.1 Geometric initialization

• Geometric initialization samples the selected-feature count from a truncated geometric distribution, then uniformly samples vectors with that count to cover feature fractions while favoring sparse models.The geometric success probability is configurable; the authors selected an empirical value based on the average number of distinct split variables across 100 decision trees fitted on random 90% subsets.Source: §4.3.1 · pp. 6–7

4.3.2 Filter-ensemble based initialization

• Filter-based initialization first samples the count S as above, then samples each feature bit with a probability dependent on its weighted filter-ensemble score.The ensemble weights w are sampled uniformly from the simplex, and higher-scoring features receive greater inclusion probability while the total count remains approximately geometric.Source: §4.3.2 · p. 7

4.3.3 Hamming-weight preserving mutation

• Hamming-weight-preserving mutation redraws erased bits with probability πB = (S + 1)/(p + 2), where S is the original vector’s Hamming weight.This avoids the feature-fraction bias associated with ordinary random bit-flip mutation.Source: §4.3.3 · p. 7

4.3.4 Filter-ensemble based mutation

• Filter-ensemble mutation makes feature-bit redraws depend on filter ranks and optimizes the weighting vector w during NSGA-II for self-adaptation.The operator approximately preserves the original feature vector’s Hamming weight.Source: §4.3.4 · p. 7

4.4 Implementation and Reproducibility

• The methods are available in the mosmafs R package, with benchmark-experiment code published on GitHub for reproducibility.The package is published on CRAN.Source: §4.4 · p. 7

5 Experiments

• The benchmark asks whether Bayesian or evolutionary search performs better, filter ensembles help, multi-objective search yields sparse solutions with little performance loss, and joint optimization improves on sequential optimization.The evaluation uses real-world binary classification tasks and compares predictive performance and feature sparsity.Source: §5 · p. 7

5.1 Benchmark Datasets

• Experiments use real-world binary classification datasets with balanced classes, numeric features, no missing values, and varied dimensionality; regression is not evaluated.Dataset dimensionality ranges from 30 to 10,937 features.Source: §5.1 · pp. 7–8

5.2 Learning Algorithms and their Hyperparameters

• The benchmark tunes Gaussian-kernel SVM, xgboost, and kernelized k-nearest neighbors, selected as distinct learning paradigms that respond to hyperparameter tuning.The learners differ in their response to dimensionality: kknn is vulnerable to it, SVM is regularized, and xgboost can implicitly select features.Source: §5.2 · p. 8

5.3 Algorithms

• Comparisons cover NSGA-II and Bayesian multi-objective methods with and without filter ensembles, single-objective Bayesian baselines, and sequential variants that separate tuning from feature selection.The sequential GA variant tunes hyperparameters first, then performs multi-objective feature selection; the BO variant tunes hyperparameters and filter weights before constructing a sparsity–performance Pareto set.Source: §5.3 · p. 9

5.4 Evaluation

• Each method receives up to 2,000 model evaluations, and nested resampling estimates generalization by evaluating solutions on held-out test folds across 10 outer cross-validation runs.Multi-objective performance is assessed using generalization dominated hypervolume from the optimization-set Pareto set evaluated on test data.Source: §5.4 · p. 10

6.1 Evolutionary vs. Bayesian Optimization, Effect of Filter Ensembles

• BO-MO-FE usually has a slight absolute performance advantage over GA-MO-FE, but incurs computational overhead that matters more when model evaluations are fast.That overhead becomes less relevant for large datasets or slow models.Source: §6.1 · p. 10, Figure 4 · p. 11

6.2 Multi-Objective vs. Single-Objective

• Multi-objective methods often find much sparser models than the single-objective baseline while giving up very little predictive performance.Among the best-performing configurations, multi-objective methods are on par with single-objective methods except for kknn; Bayesian methods outperform evolutionary methods.Source: §6.2 · p. 10

6.3 Simultaneous Hyperparameter Tuning and Feature Selection

• Joint optimization yields a considerable domHVgen advantage over corresponding non-joint methods in most cases, especially over BO-SO-FE-NJ.The authors suggest the advantage depends on how strongly hyperparameter performance interacts with sparsity.Source: §6.3 · p. 10, Table 4 · p. 12

6.1 Evolutionary vs. Bayesian Optimization, Effect of Filter Ensembles

• BO-MO-FE significantly outperforms all other shown methods by global domHVgen rank after 2,000 evaluations, though the absolute difference is small.The optimization trace suggests a filter-ensemble advantage, but it is not statistically significant after 2,000 evaluations.Source: §6.1 · p. 10, Figure 3 · p. 11

• Both evolutionary and model-based methods efficiently support model-agnostic joint multi-objective tuning and feature selection, and simultaneous optimization outperforms separate execution.Source: §7 · p. 13
• The model-based adaptation of AutoML tuning over filter type and included fraction works, but parameterized filter ensembles outperform it.Source: §7 · p. 13
• The specialized NSGA-II nearly matches, but does not reach, the best Bayesian optimization approach while adding less overhead when model evaluations are cheap.Source: §7 · p. 13
• Use Bayesian optimization with parameterized filter ensembles when evaluations are expensive and near-optimal configurations matter; NSGA-II suits cheap evaluations when marginal performance degradation is acceptable.Source: §7 · p. 13

Supplementary Material

• Figure 7 clusters filters by average cross-dataset L1 distance between rank-scaled scores; univariate.model.score varied unusually between repeated evaluations and was excluded.Source: Figure 7 · p. 16

NSGA-II Ablation Study

• The basic NSGA-II performed far worse than the specialized variants because its initial population did not cover the feature-fraction dimension well.Source: NSGA-II Ablation Study · p. 17
• Table 6 compares joint BO-MO-FE and GA-MO-FE with methods that tune and select features separately, reporting domHVgen after 2000 evaluations.Source: Table 6 · p. 18
• Figure 8 reports per-dataset, per-algorithm ranks averaged across datasets, with higher rank values indicating better performance.Source: Figure 8 · p. 19
• Figures 9 and 10 compare performance and included-feature fraction for BO-SO, BO-MO-FE, and GA-MO-FE after 2000 evaluations, averaged over 10 outer cross-validation runs.Source: Figure 9 · p. 19, Figure 10 · p. 20