Source-linked AI summary
Consistency-based Semi-supervised Active Learning: Towards Minimizing Labeling Cost
Mingfei Gao, Zizhao Zhang, Guo Yu, Sercan O. Arik, Larry S. Davis, Tomas Pfister
TL;DR
The paper addresses costly labeling and cold-start limitations in pool-based active learning. It unifies semi-supervised training with consistency-based selection, and experiments on CIFAR-10, CIFAR-100, and ImageNet report superior performance with limited labeled data while introducing a measure for selecting the starting point.
Problem
Conventional pool-based active learning largely ignores unlabeled data during training, while small initial labeled sets can cause biased models and poor subsequent selections.
Method
The framework trains with labeled and unlabeled data using semi-supervised learning and selects samples with low prediction consistency across augmentations.
Results
The method outperforms existing active-learning methods and alternative active-learning and semi-supervised-learning combinations on CIFAR-10, CIFAR-100, and ImageNet with limited labels.
Takeaways & Limitations
Consistency-based selection provides a unified way to leverage unlabeled data during training and prioritize samples for labeling, while the proposed measure helps determine a proper starting size.
Takeaways & Limitations
The study mainly uses MixMatch and notes that broader analyses with extensive SSL methods are desirable.
Abstract
from arXiv · showhide
Active learning (AL) combines data labeling and model training to minimize the labeling cost by prioritizing the selection of high value data that can best improve model performance. In pool-based active learning, accessible unlabeled data are not used for model training in most conventional methods. Here, we propose to unify unlabeled sample selection and model training towards minimizing labeling cost, and make two contributions towards that end. First, we exploit both labeled and unlabeled data using semi-supervised learning (SSL) to distill information from unlabeled data during the training stage. Second, we propose a consistency-based sample selection metric that is coherent with the training objective such that the selected samples are effective at improving model performance. We conduct extensive experiments on image classification tasks. The experimental results on CIFAR-10, CIFAR-100 and ImageNet demonstrate the superior performance of our proposed method with limited labeled data, compared to the existing methods and the alternative AL and SSL combinations. Additionally, we study an important yet under-explored problem -- "When can we start learning-based AL selection?". We propose a measure that is empirically correlated with the AL target loss and is potentially useful for determining the proper starting point of learning-based AL methods.
1 Introduction
The paper targets costly labeling and the cold-start problem in pool-based active learning by integrating unlabeled-data training with consistency-based sample selection.
- Motivation: Large labeled datasets can be expensive to acquire, motivating active learning methods that improve models with fewer labeled examples.This concern is especially acute when expert annotators are required.
- Active learning setting: Pool-based active learning iteratively selects unlabeled batches for labeling and retrains the model to maximize performance under limited labeling cost.
- Cold start: Insufficient starting sets cause biased models and poor later selections, creating the cold-start problem and potentially underperforming uniform sampling.
- Motivation: Semi-supervised learning can exploit abundant unlabeled data during early active-learning cycles, but relatively little active-learning research combines the two.
- Proposed framework: The framework selects samples whose predictions vary across augmentations because low consistency suggests the model cannot distill useful information from them.
- Contributions: The contributions unify semi-supervised training and sample selection, introduce a consistency metric balancing uncertainty and diversity, and propose a measure for choosing the starting size.
2 Related Work
Related work spans uncertainty-, diversity-, and model-performance-based active learning, while only a small number of studies combine active learning with semi-supervised learning.
- Active learning: Active-learning methods commonly use uncertainty, diversity, or model-performance-change criteria to select samples.
- Active learning: Uncertainty methods often rely on entropy, margin, or decision-boundary heuristics, while recent work directly learns target loss during training.
- Semi-supervised active learning: Only a few recent studies combine active learning and semi-supervised learning, with reported benefits in speech understanding and pedestrian detection.
- Agreement-based selection: The proposed method relates to agreement-based active learning by selecting samples whose predictions disagree most across augmentations.
3 Consistency-based Semi-supervised AL
The method trains on labeled and unlabeled data with a consistency objective, then selects batches whose augmented predictions are most inconsistent for human labeling.
- Training: The framework minimizes supervised and unsupervised losses jointly, using labeled data for cross-entropy and unlabeled data for consistency-based training.
- Algorithm: The active-learning procedure begins with a uniformly sampled labeled set, alternates selection, labeling, and pool updates, and retrains across cycles.
- Consistency objective: Consistency-based SSL encourages similar predictions for inputs and augmentations that preserve perceptual content.
- Selection rationale: The selection rationale is that highly inconsistent samples are difficult for the unsupervised loss to minimize and may benefit from human annotations.
- Selection metric: The per-sample metric quantifies prediction inconsistency across the original input and a random set of augmented versions.
- Batch selection: Batch selection maximizes the aggregate inconsistency metric over a fixed-size subset, after which selected samples are labeled and removed from the unlabeled pool.
4 Experiments
Experiments on CIFAR-10, CIFAR-100, and ImageNet compare consistency-based selection with SSL and supervised AL baselines. The method improves accuracy with limited labels and its analyses connect performance to uncertainty handling, diversity, class-error alignment, and cold-start behavior.
- Experimental setup: Experiments compare the proposed method with naive AL and SSL combinations, recent supervised AL methods, and analyses of selected-sample attributes.Evaluations use CIFAR-10, CIFAR-100, and ImageNet; reported curves average multiple trials.
- Comparison with selection baselines under SSL: 2.5% higher accuracy than uniform and 1.5% higher than k-center with 250 labeled CIFAR-10 images under SSL.Using 2K labels matches uniform selection with 4K labels, halving labeled-data requirements for similar performance.
- Comparison with supervised AL methods: 22.62% and 12.28% accuracy improvements over VAAL at 40% labeled data on CIFAR-100 and ImageNet, respectively.The proposed method significantly outperforms supervised AL methods across all AL cycles on both datasets.
- Comparison with supervised AL methods: 39.24% higher accuracy than k-center when 150 CIFAR-10 samples are labeled, compared with baselines trained only on labeled data.Table 2 compares SSL training for the proposed method against supervised baseline training with very few labels.
- Analyses of consistency-based selection: Consistency selects highly uncertain samples without necessarily choosing the most uncertain, while matching entropy-based selection in class-error correlation and diversity.Analyses report better detection of overconfident misclassifications, diverse selections, and class distributions correlated with per-class error.
- Analyses of consistency-based selection: Uncertainty-based selection suffers from cold-start bias when the learned decision boundary is far from the expected boundary, whereas uniform sampling explores the whole space.The toy two-moons analysis links biased uncertainty selections to sub-optimal predictive performance.
5 When can we start learning-based AL selection?
The section examines cold-start failure in learning-based active learning and proposes tracking cross-entropy between predicted and assumed label distributions to identify a suitable starting size.
- Cold-start failure: Extremely small initial labeled sets can produce biased models and poor subsequent sample selection, causing cold-start failure.The learned decision boundaries may be far from the real boundaries, making model-based selection unreliable.
- Cold-start failure: Starting with K0 = 50 underperforms starting with K0 = 100 when both models reach 150 labeled samples on CIFAR-10.The comparison illustrates that initial-set size affects later active-learning accuracy.
- Cold-start failure: Semi-supervised learning supports initiating learning-based sample selection from a smaller labeled start set, but extremely small sets can still yield poorly learned boundaries.A sufficiently large validation set could relieve this issue, but such a set typically does not exist in practice.
- Start-size criterion: The unavailable target loss over the entire training set motivates an approximation based on bounds involving H[p(Y ), p( ˆY )].This quantity can be tracked without labeling all training data by specifying a distribution for Y.
- Start-size criterion: H[p(Y ), p( ˆY )] strongly correlates with the target loss in both semi-supervised and supervised settings.The measure can guide expanding the start set until consecutive differences fall within a predefined threshold.
- Start-size criterion: In the SSL setting, 100 or 150 labeled samples may serve as the CIFAR-10 start set because the tracked value essentially ceases changing.The exact optimal start size remains an exploratory question rather than a fully determined rule.
6 Weaknesses of our method
The section reports limitations of the method’s supervised application and identifies incomplete analysis of SSL variants and exact start-size selection.
- Supervised setting: With 3,000 labels after four CIFAR-10 AL cycles, consistency sampling reaches 82.75% accuracy, comparable to k-center at 81.70% and entropy at 82.67%.The supervised experiment does not show significant improvement over baseline selection metrics.
- Supervised setting: The selection metric’s direct application in supervised learning is discouraged because it provides no significant improvement over baseline metrics.The reported supervised results use 1,000 labeled samples initially and average results over five trials.
- Scope of analysis: Comprehensive analyses across extensive SSL methods are still desirable to clarify the approach’s advantages and disadvantages.MixMatch is mainly used as the target model in the reported work.
- Scope of analysis: The proposed start-size measure is strongly correlated with AL target loss, but the optimal start size is not determined exactly.This leaves start-size selection as an exploratory rather than fully resolved problem.
7 Conclusion
The paper presents a consistency-based semi-supervised active learning framework that uses unlabeled data during training and selection. It also proposes a measure for determining when learning-based selection can begin.
- The framework leverages unsupervised information from unlabeled data during model training and sample selection.
- The proposed semi-supervised active learning method outperforms existing active learning methods and alternative SSL–AL combinations.
- The selection metric implicitly balances uncertainty and diversity when choosing samples for labeling.
- The paper introduces a measure for determining the proper starting size of learning-based active learning.
- The proposed measure correlates well with AL target loss and may evaluate target models without extra labeling effort.