Source-linked AI summary
Meta-Learning for Classifier Selection in Image Datasets: A Feature-Driven Framework for Accuracy Prediction
Zahra Nabizadeh_Shahre_Babak, Farzaneh Koohestani, Nader Karimi, Shahram Shirani, Shadrokh Samavi
TL;DR
Classifier selection for image datasets is difficult because dataset characteristics vary and no single model is universally optimal. The paper uses dataset-complexity meta-features, feature selection, regression, and classifier grouping to predict performance without exhaustive evaluation, achieving strong ranking accuracy across diverse datasets.
Problem
Selecting classifiers for image datasets is challenging because dataset characteristics vary, while exhaustive training and evaluation is computationally expensive.
Method
The framework extracts and selects dataset-complexity meta-features, predicts classifier accuracies with a meta-model, and groups classifiers with similar performance patterns.
Results
86.15% mean prediction accuracy across 15 ranking levels was achieved across 56 diverse image datasets, with rank 2 reaching 100% accuracy.
Takeaways & Limitations
The framework provides a scalable and interpretable approach for recommending classifiers while reducing the need for exhaustive training and evaluation.
Takeaways & Limitations
Pre-trained models may not generalize well to domain-specific datasets whose feature distributions differ from natural images.
Abstract
from arXiv · showhide
No Free Lunch theorem implies that any performance gains achieved by a classifier on a particular image distribution are necessarily offset by a loss of performance over the set of all possible problems; thus, no single model is universally optimal. Selecting the most suitable classifier for image datasets is a critical yet challenging task due to the intrinsic complexity and diversity of images. This paper proposes a meta-learning framework that leverages a comprehensive set of meta-features capturing dataset complexity to predict classifier performance without exhaustive training. By extracting and selecting features using methods such as autoencoders, pre-trained networks, and dimensionality reduction techniques, we train regression models to efficiently estimate classifier accuracies. Additionally, clustering techniques are employed to group classifiers with similar performance patterns, simplifying the recommendation process. The datasets used span a wide range of concepts, including nature, animals, numbers, motorcycles, medical images, and human bodies, to ensure broad generalization. Evaluated on 56 diverse image datasets, our approach achieves an average ranking prediction accuracy exceeding 86%, demonstrating its effectiveness in guiding model selection. This scalable and interpretable framework provides a practical solution to improve classification performance while reducing computational costs.
1 Introduction
Image-dataset complexity makes classifier selection difficult, while existing complexity measures and meta-features support data-driven model recommendations. The paper addresses this gap with a meta-learning framework that selects relevant descriptors to predict classifier performance.
- Image-specific measures include HOG for edge structure, LBP for texture, Msep for class separability, and Mvar for within-class variability.
- The framework targets computational and interpretability challenges caused by high-dimensional image data, real-world variability, and opaque deep models.
- The paper builds a meta-learning framework that maps image-dataset complexity measures to predicted classifier performance for more efficient selection.
- The framework generates a meta-dataset of image-dataset characteristics and trains a meta-model to predict classifier accuracy on new datasets.
- Feature-selection methods identify the most relevant descriptors from categorized meta-features for classifier-performance prediction.
2 Formulation of Decision-Making as a Machine Learning Problem
The paper formulates classifier selection as a pipeline that standardizes images, extracts and selects dataset meta-features, predicts classifier accuracies, and chooses the highest-scoring model.
- 2 Formulation of Decision-Making as a Machine Learning Problem: Classifier selection is formulated as meta-learning that maps dataset characteristics to the performance of candidate algorithms.
- 2.1 Preprocessing: Images are resized to a fixed resolution before direct image operations or transformation into feature vectors, producing standardized inputs for meta-feature extraction.
- 2.2 Feature Extraction and Meta-Dataset Creation: Meta-features extracted from resized images or vectors describe class separability, feature redundancy, intrinsic dimensionality, and statistical structure.
- 2.3 Feature Selection: Feature selection reduces meta-dataset dimensionality by retaining the subset of meta-features most predictive of classifier performance.
- 2.4 Training a Meta-Model: The meta-model learns a regression relationship from selected meta-features to the predicted accuracy of each classifier.
- 2.5 Select Proper Classifier: For a new dataset, the trained model predicts each classifier’s accuracy and selects the classifier with the highest prediction.
3 Proposed Method
The proposed framework predicts classifier accuracy from image-dataset meta-features, using image/vector representations, dimensionality reduction, feature selection, and classifier-specific regression models.
- The framework predicts classifier accuracy from dataset-specific features without extensive training, supporting automated model selection.It also aims to relate dataset characteristics to classifier performance and improve interpretability.
- Images are converted into vectors through flattening or learned representations from autoencoders and pre-trained networks such as VGG19.Flattening is efficient but loses spatial and hierarchical structure, whereas neural feature extraction captures higher-level patterns and semantic information.
- PCA and t-SNE reduce high-dimensional image representations to improve computational efficiency while retaining important dataset characteristics.PCA preserves major variance through linear projections, while t-SNE preserves local structure through nonlinear mapping.
- The meta-dataset pairs extracted features with accuracy scores from multiple classifiers, enabling regression models to estimate classifier performance.The framework outputs predicted accuracies, individual classifier rankings, and classifier-group rankings.
- Meta-features capture image structure, class separability, distributional properties, and graph-based characteristics to characterize dataset complexity.The categories include data complexity, class separability and overlap, and geometric and topological properties.
4 Experimental Results
Experiments use 56 image datasets and compare multiple image-vector representations and dimensionality-reduction settings to identify features most correlated with classifier performance.
- 56 image datasets spanning medicine, nature, and industry are divided into 40 training datasets and 16 test datasets.Images are resized to a fixed resolution for computational uniformity.
- Four vector-representation sets are produced by combining two neural models with dimensionality reduction through PCA or t-SNE.PCA retains 90% of reconstructed variance, while t-SNE reduces features to dimension 10.
- AutoEncoder features with transfer learning combined with PCA show stronger correlations with classifier performance than the compared feature sets.Correlations are assessed using Pearson, Spearman, and Kendall measures, with results reported in Table 2.
4.3 Compare Feature Selection
The study compares correlation analysis, SHAP, and RFECV for selecting meta-features and finds substantial overlap alongside method-specific emphasis on image features.
- 26 features are shared between correlation analysis and RFECV, 21 between correlation analysis and SHAP, and 12 across all three methods.
- Feature-selection methods substantially agree, indicating that regression models prioritize features associated with classifier-accuracy trends.
- Texture and edge features are absent from correlation-selected top features but appear among features selected by RFECV or SHAP.
4.4 Classifier-Based Accuracy Prediction
The framework trains separate regression models for each classifier because classifiers use different learning mechanisms, then evaluates prediction errors with MSE and MAPE.
- Separate regression models are trained for each classifier because the classifiers employ distinct learning mechanisms.The classifier set includes models such as logistic regression, Ridge, SVM, LDA, QDA, and Naive Bayes.
- Classifier-specific prediction errors on the test set are reported using mean squared error (MSE) and mean absolute percentage error (MAPE).
4.5 Select Best Model
The framework ranks classifiers by predicted accuracy, but using all features achieves only 50% ranking accuracy, compared with 40% for correlation-selected features.
- 50% ranking accuracy is achieved using all features, compared with 40% when correlation-based selected features are used.
4.6 Select Best Group Model
The proposed alternative clusters classifiers with similar training performance before using regression-based accuracy predictions to refine recommendations.
- K-means applied to 15 classifiers identifies four performance-based clusters using the elbow method to select the cluster count.
- The reported clusters group RF, DT, ET, and GB; SVM, KNN, MLP3, MLP5, and QDA; and NB, LDA, LR, RC, and ABC.
- The evaluation compares true test-set rankings with rankings produced from regression-predicted classifier accuracies.
- Clustering classifiers before ranking reduces sensitivity to small accuracy differences and improves interpretability by grouping methods with similar strengths.
- The hybrid clustering-and-regression approach is intended to balance ranking precision and practicality for diverse image-classification tasks.
4.7 Results
The clustering-and-regression framework predicts classifier rankings with a mean accuracy of 86.15%, performs especially well at top ranks, and favors PCA with autoencoder features.
- 86.15% mean prediction accuracy is achieved across all 15 classifier-ranking levels.
- PCA with autoencoder features achieves the best performance among pca ae, pca vgg19, tsne ae, and tsne vgg19.
- Performance depends on extracted feature counts, transfer learning with pre-training, and dataset-specific dimensionality reduction.
- Above 95% accuracy is achieved at ranks 1–3, with rank 2 reaching 100% accuracy.
- Ranks 4–10 show moderate accuracy ranging from approximately 72% to 90%, indicating greater ambiguity among mid-ranked classifiers.
- Ranks 11–15 maintain relatively stable accuracy of approximately 85%–91%.
4.8 Comparison
On image datasets, the proposed features outperform a comparable 22-feature approach, while feature reduction offers a trade-off between extraction complexity and ranking accuracy.
- The comparison applies prior 1D-dataset features to image datasets and evaluates them against the proposed image-oriented features.
- 87.60% ranking accuracy is obtained with the first 22 proposed features, versus 85.41% using the 22 features from prior work.
- Using only 12 features reduces feature-extraction complexity by nearly 70% but lowers accuracy by 5.32%, from 86.15% to 81.56%.
- Removing 13 decision-tree-derived features decreases accuracy slightly from 86.15% to 85.0%.
5 Comparison with Related Work
Prior studies examine classifier behavior in specialized domains, under noise, or across datasets, whereas this work predicts classifier performance for heterogeneous image datasets from dataset characteristics.
- Prior work studies image classification in controlled industrial settings, emphasizing accuracy within a fixed application domain rather than cross-domain model selection.
- Noise-focused research evaluates how classifier performance degrades under controlled perturbations, emphasizing robustness and stability in corrupted images.
- Cross-dataset analyses show that classifier performance varies with image distributions, supporting the absence of a universally superior classifier.
- Research on software defect prediction similarly links classifier effectiveness to dataset properties, but uses tabular rather than image datasets.
- The proposed framework differs by predicting classifier performance directly from dataset characteristics across heterogeneous image datasets.It is designed for classifier recommendation rather than single-domain optimization, robustness testing, or descriptive cross-dataset comparison.
6 Conclusion
The paper presents a meta-learning framework that predicts classifier accuracies from selected image-dataset complexity features without exhaustive evaluation. Across 56 diverse image datasets, it reports accurate ranking prediction and a scalable, interpretable recommendation approach.
- The framework predicts classifier accuracies from extracted, selected image-dataset complexity measures without exhaustive training and evaluation of every candidate model.
- Autoencoders, pre-trained deep networks, PCA, and t-SNE represent image datasets, while classifier-specific regression models identify relevant performance features.
- Clustering classifiers by performance similarity groups models with comparable accuracies, simplifying decisions and improving interpretability and robustness.
- 56 diverse image datasets support high average accuracy in predicting classifier rankings, especially for identifying top-performing models.
- The resulting method is presented as scalable, interpretable, and effective for classifier recommendation, with future extensions proposed for other data types and advanced meta-features.