Source-linked AI summary
META-DES: A Dynamic Ensemble Selection Framework using Meta-Learning
Rafael M. O. Cruz, Robert Sabourin, George D. C. Cavalcanti, Tsang Ing Ren
TL;DR
The paper addresses the insufficiency of relying on local accuracy estimates alone to assess classifier competence. It proposes META-DES, a meta-learning framework using five meta-feature sets, and reports the highest classification accuracy in the majority of datasets.
Problem
Local accuracy estimates alone are insufficient for achieving results close to the Oracle.
Method
META-DES uses five sets of meta-features representing different dynamic selection criteria to train a meta-classifier for estimating classifier competence.
Results
META-DES achieved the highest classification accuracy in the majority of datasets.
Takeaways & Limitations
The proposed meta-learning framework improves dynamic ensemble selection performance across the evaluated classification problems.
Takeaways & Limitations
Pre-computing ensembles greatly increases overall system complexity.
Abstract
from arXiv · showhide
Dynamic ensemble selection systems work by estimating the level of competence of each classifier from a pool of classifiers. Only the most competent ones are selected to classify a given test sample. This is achieved by defining a criterion to measure the level of competence of a base classifier, such as, its accuracy in local regions of the feature space around the query instance. However, using only one criterion about the behavior of a base classifier is not sufficient to accurately estimate its level of competence. In this paper, we present a novel dynamic ensemble selection framework using meta-learning. We propose five distinct sets of meta-features, each one corresponding to a different criterion to measure the level of competence of a classifier for the classification of input samples. The meta-features are extracted from the training data and used to train a meta-classifier to predict whether or not a base classifier is competent enough to classify an input instance. During the generalization phase, the meta-features are extracted from the query instance and passed down as input to the meta-classifier. The meta-classifier estimates, whether a base classifier is competent enough to be added to the ensemble. Experiments are conducted over several small sample size classification problems, i.e., problems with a high degree of uncertainty due to the lack of training data. Experimental results show the proposed meta-learning framework greatly improves classification accuracy when compared against current state-of-the-art dynamic ensemble selection techniques.
1. Introduction
Dynamic ensemble selection estimates classifier competence locally, but the paper argues that one criterion is insufficient. META-DES uses meta-learning and five meta-feature sets to learn which classifiers are competent for each input.
- Dynamic ensemble selection: Dynamic ensemble selection chooses classifiers deemed competent for the local feature-space region surrounding each test sample.The selection retains classifiers meeting a competence level defined by a selection criterion.
- Motivation: Local accuracy estimates alone are insufficient to approach Oracle performance, motivating multiple competence criteria.The Oracle always selects a classifier that predicts the correct label when one exists.
- Motivation: Consensus-based criteria can fail when classifiers do not reach consensus and neglect base-classifier local performance.These limitations illustrate why a single competence perspective may be unreliable.
- Proposed framework: META-DES proposes five meta-feature sets, each capturing a different property of base-classifier behavior and corresponding to a dynamic selection criterion.The criteria include local-region classification performance and classifier confidence.
- Proposed framework: A meta-classifier learns from the meta-features whether each base classifier is competent enough to classify a given input.Meta-features are extracted during meta-training and from query instances during generalization.
- Evaluation: The framework was evaluated on 30 classification problems against eight state-of-the-art dynamic selection techniques and static combination methods.The evaluation emphasizes small datasets while also including some larger datasets under different conditions.
2. Classifier competence for dynamic selection
Dynamic selection chooses a sample-specific subset of classifiers by estimating competence, but its effectiveness depends on how competence and local regions are measured. Existing approaches use feature-space accuracy, decision-space similarity, or ensemble consensus, each with distinct limitations.
- Dynamic selection forms a sample-specific ensemble C′ from a classifier pool C, unlike static selection, which fixes the ensemble during training.The selected subset is intended to contain the best classifiers for a given test sample.
- The central challenge is measuring each base classifier’s competence for the query sample rather than relying only on global validation performance.Competence is treated as how much an expert is trusted for a particular classification task.
- Classifier accuracy over a local region: Feature-space methods estimate competence from local accuracy, including overall or class-specific accuracy, distance-weighted accuracy, consecutive correct classifications, and oracle-style correctness.These methods define a region around the query and apply different criteria to classifier behavior within it.
- Classifier accuracy over a local region: Local-accuracy techniques depend on the algorithm defining the competence region and can be affected by outliers, validation–test distribution differences, and insufficient information from local accuracy alone.These constraints can limit performance relative to the Oracle.
- Decision Templates: Decision-space methods compare output profiles or templates and select classifiers using similarity or performance thresholds, rather than feature-space neighborhoods.An output profile records the decisions produced by the base classifiers for an instance.
- Extent of Consensus or confidence: Decision-space methods avoid dependence on feature-space region quality but may consider only global information and neglect each classifier’s local expertise.Consensus-based approaches also precompute ensembles, increasing overall system complexity.
3.1. Problem definition
The paper formulates dynamic selection as a meta-classification problem: predict whether each base classifier is competent for a query instance. A learned meta-classifier combines multiple competence criteria to make the selection decision.
- The meta-problem predicts whether a base classifier is competent or incompetent for classifying a given test sample.Its two meta-classes are “competent” and “incompetent.”
- Each meta-feature represents a different criterion for measuring classifier competence, and the vector v_i,j encodes classifier behavior relative to input x_j.The representation transfers information from the classification environment into the meta-classification environment.
- A meta-classifier λ is trained on the meta-feature vectors to predict whether classifier c_i will correctly classify x_j.The learned prediction determines whether the classifier is competent enough for the sample.
- The proposed system differs from existing dynamic selection techniques by using multiple competence criteria and learning the selection rule from training data.
3.2. The proposed META-DES
META-DES combines five competence criteria through meta-learning to decide which classifiers should join the ensemble for each test sample. Its pipeline generates a classifier pool, trains a meta-classifier, and applies dynamic selection during generalization.
- Framework overview: META-DES has three phases: classifier overproduction, meta-training, and generalization with a dynamically selected ensemble.The generalization phase applies majority voting to the selected ensemble to produce the test label.
- Meta-training: A separate meta-training dataset Tλ is used to extract feature vectors and train selector λ, helping prevent overfitting.The stored meta-feature vectors are divided into validation and training data before λ is trained.
- Generalization: During generalization, λ predicts whether each base classifier is competent enough to enter C′, whose members jointly determine the output label.The query instance supplies the meta-information used for each classifier-level decision.
- Sample selection: Meta-training focuses on samples whose pool consensus H(xj,trainλ, C) falls below the consensus threshold hC.For these samples, the method computes a competence region, output-profile neighbors, and classifier-specific meta-features.
- Meta-feature extraction: The five feature sets represent distinct competence criteria, allowing other features to contribute when one criterion is imprecise or confidence is low.The framework uses local-region information, similar output profiles, and input-sample confidence to estimate competence.
4. Experiments
The experiments evaluate META-DES on 30 datasets using repeated train, dynamic-selection, and test partitions, with parameters tuned on a subset of datasets. Results compare it with dynamic and static ensemble-selection methods.
- Experimental protocol: 30 datasets from UCI, STATLOG, KEEL, medical, and artificial sources were used in the comparative experiments.The collection includes both ill-defined problems and larger databases.
- Experimental protocol: Each replication allocated 50% of data to training, 25% to DSEL, and 25% to testing while preserving class priors.For META-DES, the training portion was split equally between meta-training and classifier-pool generation.
- Parameter analysis: Parameter tuning selected K = 7 and Kp = 5, while hC = 70% was statistically superior on 6 of 11 tuning datasets.Kp = 5 showed the best results on the Sonar, Ionosphere, and Liver Disorders datasets where comparisons differed significantly.
- Small-sample problems: For small-size problems including Liver Disorders, Sonar, Weaning, and Ionosphere, META-DES achieved statistically superior recognition accuracy.These results address the paper’s comparison of performance on ill-defined problems.
- Dynamic-selection comparison: META-DES was superior or equivalent to state-of-the-art DES techniques on 25 datasets, or 84% of the datasets.It achieved the highest recognition performance on 18 datasets, corresponding to 60%.
- Dynamic-selection comparison: META-DES was statistically inferior to the best state-of-the-art DES result on 5 datasets, representing 16% of the datasets.The datasets were Ecoli, Heart, Vehicle, Banana, and Lithuanian.
- Static-method comparison: META-DES achieved the highest recognition accuracy on 24 of 30 datasets against static ensemble techniques.The paper attributes this pattern to using multiple properties of the classification environment as meta-features.
5. Conclusion
META-DES is a meta-learning dynamic ensemble selection technique that uses five meta-feature sets to estimate base-classifier competence. Across 30 datasets, it achieved the highest classification accuracy in the majority and improved performance particularly on critical training-size problems.
- Contribution: META-DES combines five meta-feature sets, each representing a different dynamic selection criterion, to estimate classifier competence.The meta-features encode properties from the feature and decision spaces and train a meta-classifier to assess whether a base classifier is competent for an input sample.
- Method: The framework extracts meta-features for new test data and passes them to a meta-classifier that determines whether each base classifier should classify the sample.This separates classification and meta-classification environments while learning the classifier-selection rule.
- Experimental study: Experiments covered 30 classification datasets and compared META-DES with eight state-of-the-art dynamic selection techniques and five static combination methods.The datasets came from five repositories: UCI, KEEL, STATLOG, LKC, and ELENA.
- Results: META-DES achieved the highest classification accuracy in the majority of datasets, supported by its use of five distinct DES criteria.The authors attribute the result to considering multiple criteria during ensemble selection, so other criteria can contribute when one fails.
- Results: META-DES showed significant performance improvement on datasets with critical training-size samples.The authors relate this gain to generating several meta-feature vectors per training sample, providing more data for training the meta-classifier than current DES methods.
- Future work: Future work includes new meta-feature sets, optimization-based meta-feature selection, and evaluation of different meta-classifier training scenarios.These directions target competence estimation, meta-classifier performance, and training-scenario effects.