Source-linked AI summary
Diversity-Based Active Learning: An Evaluation of Metric Spaces for Active Learning Selection
Siddharth Chilamkur, Dorit S. Hochbaum
TL;DR
Active learning addresses the cost of obtaining labeled data, but uncertainty-focused selection can sample redundant or unrepresentative regions. This paper evaluates Greedy K-center in several metric spaces using Random Forests as the evaluator, finding that predictive probability spaces—especially with entropy weighting—usually perform best across the tested datasets.
Problem
Uncertainty-focused active learning can overselect localized outliers, noise, or redundant samples, motivating evaluation of more representative metric spaces for Greedy K-center.
Method
The study compares Greedy K-center in raw feature, LDA, predictive probability, and entropy-weighted probability spaces using Random Forest classifiers and synthetic and real-world datasets.
Results
Predictive probability spaces generally outperform the other evaluated options, with entropy weighting often producing slightly better and faster convergence while being no worse in worst-case scenarios.
Takeaways & Limitations
The probability space is the most effective tested metric space for Greedy K-center when paired with a robust Random Forest baseline, with entropy weighting balancing diversity and uncertainty.
Abstract
from arXiv · showhide
With rapid advancement over the last few years, many different methods are now widely used for classification. However, training these models requires substantial labeled data. Active Learning is a potential solution to this problem. Pool-based active learning minimizes costs by querying only the most informative samples from an unlabeled dataset. Diversity-based approaches, on the other hand, attempt to select a representative subset of the data. There are many different objectives for determining the selection process, including exact K-center, exact K-median, and Greedy K-center. In this paper, we will focus on evaluating the performance of Greedy K-center across a variety of metric spaces: the raw feature space, a Linear Discriminant Analysis (LDA) space, and a model-derived probability space (with and without entropy-based weighting). Using Random Forest classifiers as a baseline evaluator, our empirical results on synthetic and real-world datasets demonstrate that mapping unlabeled instances into a predictive probability space and weighting the result by entropy often dominates the other options for active learning selection with Greedy K-center.
I. INTRODUCTION
Active learning reduces labeling costs by querying informative samples, but uncertainty-only strategies can overfocus on redundant or noisy regions. This paper evaluates Greedy K-center across metric spaces and finds model-derived probability spaces, especially entropy-weighted ones, most effective across the tested settings.
- Motivation: Pool-based active learning iteratively scores an unlabeled pool and queries batches for labeling, reducing reliance on exhaustive manual annotation.The labeled batch is added to training data, followed by retraining until the target labeling amount or performance threshold is reached.
- Motivation: Uncertainty- and disagreement-based sampling can concentrate queries on outliers, noise, or redundant local clusters while neglecting broader data regions.
- Diversity-based selection: Diversity-based methods instead seek subsets that represent the global data distribution using objectives such as K-median and K-center.
- Greedy K-center: Exact K-center is NP-hard, while prior work found Greedy K-center empirically preferable to exact K-center and exact K-median too computationally expensive for this setting.
- Greedy K-center: Greedy K-center achieves a 2-approximation by repeatedly selecting the unlabeled point farthest from its closest selected center.
- Paper scope: The study compares raw features, LDA, predictive probability, and entropy-weighted probability spaces, using Random Forests and synthetic and real-world evaluations.
II. ALGORITHMS
The proposed algorithms retain Greedy K-center’s selection rule while changing the metric space used to measure distances between unlabeled and labeled instances.
- II. ALGORITHMS: The methods transform feature vectors into alternative metric spaces before calculating diversity for active learning selection.
- II. ALGORITHMS: Greedy K-center initializes with a randomly selected labeled set and then selects unlabeled points farthest from their nearest labeled point.
- II. ALGORITHMS: The subsequent algorithm variants differ in how they define the distance d(u,l).
A. Random Sampling (Baseline)
Random sampling provides a model-independent baseline by selecting unlabeled instances uniformly at random, offering an unbiased sample of the underlying data distribution.
- A. Random Sampling (Baseline): Random Sampling selects each unlabeled instance uniformly at random and adds it to the labeled set.
- A. Random Sampling (Baseline): Unlike heuristic methods, random sampling uses neither geometric distance nor predictive uncertainty.
- A. Random Sampling (Baseline): The method serves as a performance threshold for evaluating metric-space active learning approaches.
B. Feature Space
The Feature Space baseline applies Greedy K-center to standardized raw feature vectors using Euclidean distance, but treats every feature as equally relevant to classification.
- B. Feature Space: Feature Space defines d(u,l) as the L2 Euclidean distance between standardized raw feature vectors.
- B. Feature Space: This raw-distance baseline assigns equal importance to all features regardless of their predictive power for the target classification variable.
C. Linear Discriminant Analysis (LDA) Space
The paper evaluates LDA as a supervised lower-dimensional space for Greedy K-center selection, using class separation to guide distances. However, LDA can compress overlapping classes in complex, nonlinear distributions, limiting diversity selection.
- LDA projection: LDA projects data into a lower-dimensional latent space while maximizing between-class separation and minimizing within-class variance.Its projection matrix W is optimized using Fisher’s criterion, the ratio of between-class to within-class scatter.
- Greedy selection: At each active learning step, LDA is retrained on the currently labeled set before transforming both labeled and unlabeled instances.Greedy K-center then measures L2 distances between the resulting projections.
- Limitation: LDA can hurt Greedy K-center selection when complex, highly nonlinear class distributions force overlapping classes into a tight cluster.The compressed embedding makes meaningful diverse points harder to identify.
D. Probability Space
The probability-space method represents samples by the Random Forest’s predicted class probabilities and applies Greedy K-center to their distances. This emphasizes diversity relative to the model’s decision boundary rather than raw feature dimensions.
- Probability representation: The method represents each sample with its Random Forest predicted class-probability vector.Each vector is computed as the mean predicted class probabilities across the forest’s trees.
- Distance metric: Greedy selection measures L2 distance between the resulting probability vectors.
- Selection behavior: Probability-space diversity distinguishes confidently predicted classes from uncertain predictions near the model’s decision boundary.Thus, selection reflects predictive representation rather than only feature-space geometry.
E. Probability + Entropy Space
The hybrid method weights probability-space diversity by predictive entropy, combining separation in the model’s representation with attention to uncertain instances. It therefore favors points that are both distant from selected centers and uncertain.
- Entropy weighting: The hybrid method weights probability distance by the predictive entropy of each unlabeled point.Shannon entropy measures impurity in the model’s prediction across the C classes.
- Selection criterion: The modified Greedy criterion favors instances far from existing probability-space centers and having high uncertainty.Scaling shrinks distances for points that the model predicts with high confidence, making them less likely to be selected.
III. RESULTS
The results evaluate metric-space selection methods on synthetic and real-world datasets under multiple query budgets, using repeated runs and Random Forest-based evaluation. Random Forest was selected over a neural-network baseline because it offered robust stability at much lower computational cost.
- Experimental setup: The experiments cover both synthetic and real-world datasets, with each query selecting unlabeled instances for oracle labeling and training-set inclusion.
- Experimental setup: Metrics are averaged over 100 independent runs to account for variance and support statistical significance.
- Experimental setup: The evaluation uses two query-budget regimes: 10%–50% of training data and 100–400 total samples.The first adds 10% per step, while the second adds 25 samples per step; each run uses a new 80/20 split and initial labeled pool.
- Baseline evaluator: The baseline comparison evaluates Random Forest against a Pre-Activation Tabular ResNet neural network across six datasets.
- Baseline evaluator: Table I reports the fully supervised runtime comparison between Random Forest and the neural network on an L4 GPU.
- Baseline evaluator: The neural network requires more than 100 epochs for basic stability and similar accuracy, creating severe computational cost during repeated active-learning retraining.Convergence is monitored with early stopping using a 20% validation subset.
- Baseline evaluator: Random Forest is used for probability generation and final query evaluation because it stabilizes significantly faster than the neural network.The paper describes it as robust and stable in a fraction of the time, avoiding extended epoch loops and complex tuning.
B. Large Query Budgets (K Proportional to Dataset Size)
Across large query budgets, Probability Space consistently produced stronger active-learning performance than raw Feature Space or LDA, with entropy weighting adding a further enhancement.
- Random Forest was used both to generate predictive probabilities and to evaluate the queried batches.
- Probability Space consistently achieved higher terminal accuracy and AUC than Feature Space and LDA projections.
- Probability-space distances remained advantageous as synthetic datasets became more complex, from 1C through 4C.
- Entropy weighting provided a slight, consistent enhancement by discouraging selection of samples on which the model was already confident.
- The entropy-weighted method directed selection toward ambiguous samples while retaining Greedy K-center’s exploration of the decision boundary.
C. Small Query Budgets (K ≤400)
Under a maximum budget of 400 samples, Probability Space generally outperformed the baselines across several real-world datasets, while Madelon exposed an important low-data failure case.
- The scarcity experiments initialized with 100 random samples and added batches of 25 until reaching K = 400.
- Probability Space provided a large advantage across Wall Robot Navigation, Texture, Spambase, and Segment.
- Probability Space and Prob+Ent consistently outperformed Random, Feature Space, and LDA on final accuracy and AUC at the 400-sample threshold.
- Prob+Ent frequently achieved the highest performance rank and improved early-stage active learning without significant computational overhead.
- Exceptions: The overall Probability Space advantage had two notable exceptions across the experiments.
- Phoneme: In large-batch Phoneme runs, Feature Space reached 0.893 ± 0.008 final accuracy versus 0.889 ± 0.009 for Probability Space.
- Madelon: Madelon was the only scarcity-evaluation dataset where Probability Space underperformed all other methods.
IV. CONCLUSION
The conclusion identifies metric space as a key determinant of Greedy K-center success and favors predictive probability representations, especially with entropy weighting.
- Active learning seeks to reduce annotation costs by querying the most informative samples.
- Greedy K-center provides a good approximation for subset coverage, but its success depends on the metric space in which it operates.
- Across large-batch and extreme-scarcity experiments, predictive Probability Space generally accelerated convergence and outperformed geometric spaces.
- The Madelon exception indicates that Probability Space requires at least a foundational classification understanding to be effective.
- Entropy-weighted distances balance exploration and exploitation by reducing redundant queries of high-confidence instances.