Source-linked AI summary
Deep Active Learning over the Long Tail
Yonatan Geifman, Ran El-Yaniv
TL;DR
Deep active learning needs label-efficient methods, but existing approaches provide limited practical guidance and face difficult hyper-parameter selection. The paper introduces FF-Active, which queries farthest-first points in a trained network’s representation space, and reports clear advantages over passive learning across its evaluated setting. The method is motivated by coreset compression and is also reported to outperform traditional softmax-response sampling.
Problem
Deep active learning has sparse practical evidence, while neural networks make on-the-fly hyper-parameter selection especially difficult.
Method
FF-Active selects consecutive unlabeled-pool points by farthest-first traversal in representation-layer activation space, retraining after each acquired batch.
Results
The method shows clear and significant advantages over passive learning, while traditional softmax-response sampling is useful but inferior to the proposed method.
Takeaways & Limitations
The results indicate that considerable labeling resources can be saved when improving an initially trained model with active selection.
Takeaways & Limitations
The required error reduction may be unachievable in advance, and the proposed method leaves possible improvements such as pseudo-labeling and Monte-Carlo-dropout ensembles for future work.
Abstract
from arXiv · showhide
This paper is concerned with pool-based active learning for deep neural networks. Motivated by coreset dataset compression ideas, we present a novel active learning algorithm that queries consecutive points from the pool using farthest-first traversals in the space of neural activation over a representation layer. We show consistent and overwhelming improvement in sample complexity over passive learning (random sampling) for three datasets: MNIST, CIFAR-10, and CIFAR-100. In addition, our algorithm outperforms the traditional uncertainty sampling technique (obtained using softmax activations), and we identify cases where uncertainty sampling is only slightly better than random sampling.
1 INTRODUCTION AND RELATED WORK
Active learning seeks to reduce labeling effort, but deep-network applications face sparse evidence and difficult hyper-parameter selection. This paper studies a long-tail setting where an initially trained model is improved by selecting additional labels from an unlabeled pool.
- Motivation: Active learning aims to reduce the labels needed to train a classifier to a specified accuracy, known as label complexity.Theoretical results allow logarithmic dependence on inverse excess risk in some instances.
- Challenges: Deep active learning remains difficult because neural networks are highly sensitive to hyper-parameters while early labeled sets are small and biased.The challenge includes selecting learning rates, initialization, regularization, and architectures during acquisition.
- Problem setting: The long-tail setting assumes a reasonable model is initially trained, then additional labels are selected in phases under budget or accuracy-improvement requirements.Each phase selects a specified number of samples from a large unlabeled pool.
- Contribution: The proposed setting uses early random sampling to stabilize later active learning and enable practical improvements over passive learning.The paper frames active learning as expediting model improvements over the long tail of training.
- Related work: Uncertainty sampling prioritizes unlabeled points by model uncertainty, approximated in neural networks by Softmax Response activation.Softmax Response is described as reflecting distance from the decision boundary.
- Model representation: The paper decomposes the network into a representation mapping φ and a final classifier τ, using the second-last layer as the representation layer.This decomposition supports selecting points through internal neural activations.
2 MOTIVATION: COMPRESSION SCHEMES FOR DEEP LEARNING
The paper motivates active learning with coresets that preserve model performance while compressing data. Farthest-first traversal operates on representation-layer geometry to select diverse samples, achieving substantial CIFAR-10 compression with limited accuracy loss.
- Coresets: A coreset is a dataset subsample used as a proxy for the full dataset while producing qualitatively similar algorithmic output.Coresets support efficient approximation by running the same algorithm on fewer inputs.
- Algorithm: The compression algorithm is implemented as Farthest-First Compression (FF-Comp).Its traversal is also known as the Gonzalez algorithm and provides an efficient 2-approximation for k-center clustering.
- Representation geometry: Representation-layer geometry is useful for coresets because semantically similar complex inputs tend to be closer on learned manifolds.The paper focuses on internal representations rather than raw image or sound spaces.
- FF-Comp: Farthest-first compression constructs class-specific coresets by greedily selecting points farthest from already selected representation vectors.The method uses Euclidean distance and selects up to c points in a stratified manner.
- Results: 50% compression on CIFAR-10 reduced accuracy from 93.23% to 91.73%, while random 50% compression reduced accuracy by 3.1%.Random subsampling required 64% of the data, or 7,000 additional labeled points, to reach 91.73% accuracy.
3 DEEP ACTIVE LEARNING WITH CORESETS
The active algorithm transfers the coreset principle to an unlabeled pool by repeatedly selecting farthest points in representation space. After each batch is labeled, the model is retrained until the budget or desired accuracy is reached.
- Active-learning loop: At each active stage, the algorithm selects a batch of b points from the current unlabeled pool and requests their labels.The labeled set grows from the initial training set, while the model is retrained after each acquisition.
- Design choice: Farthest-first active learning applies coreset selection without class stratification because the pool is unlabeled.The basic algorithm intentionally omits possible enhancements such as pseudo-labeling.
- Selection rule: FF-Active greedily chooses each new point maximizing its minimum representation-space distance from labeled and already selected points.The procedure adds the selected batch to the labeled set and retrains the model at every iteration.
4 EXPERIMENTS
Experiments compare FF-Active with softmax-response uncertainty sampling and random sampling on MNIST, CIFAR-10, CIFAR-100, and an inflated CIFAR-100 pool. FF-Active consistently outperforms Random, with widening advantages in CIFAR experiments and especially strong results on the larger synthetic pool.
- Experimental setup: FF-Active, softmax response, and Random were compared across MNIST, CIFAR-10, and CIFAR-100 experiments.The experiments evaluate active and passive sampling strategies across three standard datasets.
- CIFAR-100: FF-Active consistently outperformed Random on CIFAR-100, while matching softmax response initially before softmax-response performance degraded.The CIFAR-100 learning curves show a consistent advantage over Random and near-identical early performance with SR.
- Cross-dataset comparison: The accuracy-improvement slope was larger for CIFAR-100 than CIFAR-10, and FF-Active’s gap over Random increased in both datasets.These trends motivated testing a substantially larger unlabeled pool.
- Longer tail: The synthetic long-tail experiment inflated the CIFAR-100 pool threefold to 150,000 unlabeled images.Bootstrap sampling with replacement was used to construct the larger pool under the same experimental setting.
- Longer tail: On the inflated CIFAR-100 pool, FF-Active strongly dominated Random and dominated softmax response through most of the active-learning session.The larger-pool experiment was designed to model prolonged labeling over a longer tail.
5 INSPECTING OF FF AND SR OVER A SMALL SYNTHETIC EXAMPLE
The synthetic example contrasts FF-Active’s systematic exploration of underrepresented regions with SR’s refinement near the decision boundary. FF-Active identifies the best model with far fewer queries in this early-stage simulation, though the paper leaves later-stage behavior for future work.
- Querying strategies: FF-Active explores the least-represented point under the current coreset, whereas SR concentrates on regions around the decision boundary.The two querying functions implement systematic exploration and boundary-focused exploitation, respectively.
- Synthetic comparison: FF-Active nearly identified the best model with 6 queries, while SR remained far from it after 30 queries.This comparison comes from an early-stage synthetic simulation.
- Interpretation: The simulation indicates that FF-Active captures the problem’s geometry using far fewer queries than SR.The authors caution that the simulation emphasizes an early active-learning stage rather than the later long-tail stages.
6 CONCLUDING REMARKS
The paper concludes that its coreset-inspired active-learning method provides a practical advantage over passive learning when improving an already trained deep model. It also identifies extensions and open compression questions for future work.
- The method targets settings where an initial reasonable model already exists, enabling active learning to save considerable labeling resources during later improvement.
- The proposed active-learning algorithm achieves a clear and significant advantage over passive learning for deep networks.
- Figure 5 simulates FF-Active and SR on a two-dimensional example.
- Its querying function uses model-based coresets built from neural activation representations and greedy farthest-first traversal.
- Pseudo-labeling and Monte-Carlo dropout are proposed as possible extensions for stratified coresets, label inference, and reduced traversal variance.
- An open question is whether datasets such as CIFAR-100 or ImageNet can be compressed to a logarithmic fraction of their size while retaining high classification performance.