Source-linked AI summary
A Benchmark and Comparison of Active Learning for Logistic Regression
Yazhou Yang, Marco Loog
TL;DR
The paper addresses how active learning methods for widely used logistic regression compare when labeled data are costly. It benchmarks state-of-the-art methods across synthetic and real-world datasets, finding that uncertainty sampling performs exceptionally well overall while random sampling often remains competitive.
Problem
Because expert labeling is costly, the paper examines which active learning methods can select useful data efficiently for logistic regression.
Method
The paper benchmarks six categories containing nine active learners in pool-based, myopic binary classification across three synthetic and 44 real-world datasets.
Results
Uncertainty sampling performs exceptionally well overall, while random sampling is not overwhelmed by individual active learning techniques in many cases.
Takeaways & Limitations
The benchmark provides empirical guidance for understanding active-learning characteristics and choosing a method for logistic regression.
Takeaways & Limitations
The study restricts active learning to pool-based, myopic binary classification and does not consider batch-mode active learning.
Abstract
from arXiv · showhide
Logistic regression is by far the most widely used classifier in real-world applications. In this paper, we benchmark the state-of-the-art active learning methods for logistic regression and discuss and illustrate their underlying characteristics. Experiments are carried out on three synthetic datasets and 44 real-world datasets, providing insight into the behaviors of these active learning methods with respect to the area of the learning curve (which plots classification accuracy as a function of the number of queried examples) and their computational costs. Surprisingly, one of the earliest and simplest suggested active learning methods, i.e., uncertainty sampling, performs exceptionally well overall. Another remarkable finding is that random sampling, which is the rudimentary baseline to improve upon, is not overwhelmed by individual active learning techniques in many cases.
1. Introduction
The paper benchmarks active learning methods built on logistic regression, motivated by the high cost of labeling large unlabeled datasets. It surveys method relationships, compares empirical performance and computational cost, and focuses on pool-based, myopic binary classification.
- Motivation: Active learning selects valuable unlabeled instances for human annotation to reduce labeling effort while maintaining performance.The motivation is that acquiring data is easier than obtaining expert labels.
- Study scope: Logistic regression is used because it is widely applied and directly provides posterior class-probability estimates exploited by many active learners.The paper gives the binary posterior model P(y_i|x_i) = 1/(1 + exp−y_iw^T x_i).
- Study scope: The benchmark uses pool-based, myopic active learning, querying one unlabeled instance at a time; batch-mode selection is excluded.This defines the principal setting and scope boundary of the study.
- Contributions: The study reviews state-of-the-art logistic-regression active learning methods and explicates relationships among them.It organizes six categories and compares nine active learners.
- Contributions: Experiments cover three artificial datasets and 44 real-world datasets to examine classification performance and computational cost.The benchmark is intended to reveal method characteristics across learning-curve behavior and computational demands.
2. Active Learning Strategies and Methods
The paper compares nine active learners across six categories in pool-based, myopic logistic-regression active learning. These methods select unlabeled instances using criteria including uncertainty, expected error or variance reduction, loss change, model change, and adaptive combinations.
- Active learning setting: Pool-based myopic active learning starts with few labeled and many unlabeled instances, queries one selected sample at a time, and repeats after manual annotation.The process stops when a stopping criterion is satisfied; batch-mode active learning is outside this study.
- Compared methods: The benchmark compares nine active learners from six categories built on logistic regression.The categories are uncertainty sampling, error reduction, variance reduction, minimum loss increase, maximum model change, and an adaptive approach.
- Uncertainty sampling: Uncertainty sampling queries instances near the current decision boundary, while entropy selects the unlabeled instance with maximum uncertainty.For binary classification, entropy and the simple margin approach are equivalent.
- Error reduction: Error-reduction methods estimate how adding a labeled instance may reduce generalization error, often treating the unlabeled pool as representative of future test data.Expected Error Reduction uses posterior probabilities before and after adding a candidate; MAXER avoids relying on the current posterior for the candidate label.
- Variance reduction: Variance-reduction methods select samples expected to reduce future model-output variance, with EVR weighting potential variance changes by current estimated class probabilities.FIVR uses a Fisher-information ratio, whereas EVR is the expected-variance-reduction method retained for comparison.
- Other strategies: MLI evaluates candidate labels through a min-max logistic-loss objective on labeled data, whereas CEER uses an optimistic criterion and also includes unlabeled data.MMC favors samples combining uncertainty with a large feature-vector norm, while AAL adaptively combines uncertainty and information density.
3. Experiments
The experiments are organized around setup, synthetic- and real-world-dataset analyses, and computational-cost comparisons.
- The experimental setup is described before analyzing synthetic datasets, real-world datasets, and computational costs.
3.1. Experimental Setting
The study evaluates active learning across synthetic and real-world binary classification datasets using learning-curve area as its performance criterion. It uses controlled splits, minimal initial labels, repeated trials, and reports dataset and feature characteristics.
- Synthetic datasets: Three binary synthetic datasets are constructed to demonstrate different active-learning behaviors.Synth2 specifically tests methods that combine informativeness with representativeness because uncertainty sampling can select poorly informative points when the boundary is initially misestimated.
- Real-world datasets: The benchmark includes 44 real-world datasets drawn from UCI, MNIST, 20 Newsgroups, and 80 binary ImageNet subsets.All datasets are preprocessed as binary classification problems and span varied sample sizes and feature dimensionalities.
- Real-world datasets: MNIST contributes three difficult binary digit-pair tasks, each subsampled to 1,500 instances with 784 pixel features.
- Real-world datasets: 20 Newsgroups contributes three binary topic tasks represented by 26,241-dimensional tf.idf vectors reduced to 500 dimensions using PCA.
- Evaluation design: The experiments use equal-size training and test splits, begin with two labeled instances—one per class—and repeat each real-world experiment 20 times.
- Evaluation design: Figure 2 reports synthetic-dataset distributions and test-set classification accuracy for each active learning method with 90% confidence intervals.
- Evaluation design: Performance is measured by the area under the learning curve, where larger values are better and the optimal score is 1.The metric captures performance across varying numbers of labeled examples rather than at one fixed labeling budget.
3.2. Analysis on synthetic datasets
Synthetic experiments show that active-learning performance depends strongly on dataset structure and learning-curve stage. Preference maps explain how sampling behavior, exploration, and model misspecification relate to these differing outcomes.
- No single active-learning method outperforms all others across the synthetic datasets.The comparison includes classification-accuracy learning curves with 90% confidence intervals.
- Preference maps visualize where each algorithm queries in feature space, using kernel-density colors from blue for low density to red for high density.The maps cover either the first queried instance or the complete learning process with exponentially downweighted later rounds.
- On Synth1, uncertainty sampling concentrates queries near the estimated decision boundary, producing poor initial performance, whereas FIVR and EVR perform well early.The boundary-focused samples can yield unstable estimates when the initial model is limited.
- Across the full Synth1 process, several strategies shift from exploratory sampling toward exploitation near the decision boundary, while others change their sampling orientation.These patterns include FIVR and EVR becoming more perpendicular to the decision boundary and EER and MLI refining it later.
- On Synth2, random sampling far surpasses nearly all methods except MLI because most active learners focus on middle clusters after an incorrect initial boundary estimate.MLI explores border clusters by incorporating representativeness, enabling it to outperform random sampling and the other methods on this artificial set.
- On the complex Synth3 dataset, random sampling outperforms every active-learning method after 6 instances are selected.The authors suggest that other methods attend to local structure and can miss the dataset’s global structure; Synth3 preference maps are omitted.
3.3. Analysis on real-world datasets
Across real-world datasets, no single active-learning algorithm dominates, but ENTROPY and EER generally perform strongly. Uncertainty sampling achieves the strongest overall comparison with random sampling, while performance varies by dataset and method.
- The analysis compares methods using ALC, average ranking, win counts, and win/tie/loss results, with paired t-tests at 95% significance.
- No single algorithm outperforms all others on every dataset.
- Uncertainty sampling achieves the highest ALC score, best average ranking, and strongest win/tie/loss performance against random sampling.
- EER outperforms MAXER and CEER, while MAXER surpasses random sampling on only 20 of 44 datasets.
- EVR slightly outperforms FIVR, but FIVR is still exceeded by random sampling on 12 datasets.
- On ImageNet, ENTROPY performs best across four measures, while all other methods are outperformed by random sampling on average ALC and average ranking.
3.4. Computational Cost Analysis
The paper evaluates the computational cost of querying 40 unlabeled samples across methods, datasets, and feature dimensionalities.
- Computational cost is assessed for selecting 40 unlabeled samples on eight datasets varying in instance counts and feature dimensionalities.
- Random sampling, ENTROPY, and MMC are identified as the most efficient methods.
4. Discussion and Conclusion
The discussion concludes that ENTROPY is the most promising method on average, while active learning remains constrained by computational costs and the risk of underperforming random sampling.
- The study compares state-of-the-art active-learning methods for logistic regression and characterizes their similarities and differences.
- ENTROPY is deemed the most promising method on average, despite its simplistic and short-sighted selection criterion.
- ENTROPY outperforms min-max view, variance-reduction, and maximum-model-change methods in the experiments.
- MLI shows an advantage in querying representative instances on the Synth2 synthetic dataset.
- Expected-error-reduction, variance-reduction, and MLI methods incur computational costs that motivate work on speeding them up.
- No individual active learner consistently avoids performing worse than random sampling across real-world datasets.
Appendix
The appendix continues the paper’s performance-comparison tables for the Area under the Learning Curve.
- The appendix contains continued performance-comparison tables for Area under the Learning Curve accuracy.
- The appendix table identifies higher scores as better performance and reports comparable competitors alongside dataset results.
- Table A.1 continues from the previous page.