Source-linked AI summary
Active Learning by Feature Mixing
Amin Parvaneh, Ehsan Abbasnejad, Damien Teney, Reza Haffari, Anton van den Hengel, Javen Qinfeng Shi
TL;DR
Active learning must select valuable annotations from unlabelled data, especially when data are high-dimensional or labels are scarce. ALFA-Mix detects prediction inconsistencies in labelled–unlabelled feature interpolations, and the paper reports broad benchmark gains, particularly in low-data settings. The authors note that evaluations are commonly simulated and that real-world evaluation could benefit the method.
Problem
Selecting the most valuable unlabelled examples for annotation is challenging with high-dimensional data and limited labelled examples.
Source: §1 · pp. 1–2Method
ALFA-Mix interpolates latent representations of labelled and unlabelled instances and uses prediction changes to identify candidates, with anchors and clustering supporting efficient, diverse selection.
Source: §1 · p. 2Results
Across evaluated rounds, ALFA-Mix surpassed other baselines in around 60% of rounds and produced state-of-the-art results across image, non-image, and video datasets.
Source: §5 · p. 8Takeaways & Limitations
The reported gains are greatest when the labelled set is small and the labeling budget is limited.
Source: §5 · p. 8The authors note that active-learning evaluation is commonly simulated and that ALFA-Mix could benefit from real-world evaluations.
Source: §5 · p. 9
Abstract
from arXiv · showhide
The promise of active learning (AL) is to reduce labelling costs by selecting the most valuable examples to annotate from a pool of unlabelled data. Identifying these examples is especially challenging with high-dimensional data (e.g. images, videos) and in low-data regimes. In this paper, we propose a novel method for batch AL called ALFA-Mix. We identify unlabelled instances with sufficiently-distinct features by seeking inconsistencies in predictions resulting from interventions on their representations. We construct interpolations between representations of labelled and unlabelled instances then examine the predicted labels. We show that inconsistencies in these predictions help discovering features that the model is unable to recognise in the unlabelled instances. We derive an efficient implementation based on a closed-form solution to the optimal interpolation causing changes in predictions. Our method outperforms all recent AL approaches in 30 different settings on 12 benchmarks of images, videos, and non-visual data. The improvements are especially significant in low-data regimes and on self-trained vision transformers, where ALFA-Mix outperforms the state-of-the-art in 59% and 43% of the experiments respectively.
1. Introduction
Batch active learning must choose valuable labels under fixed per-round budgets, a challenge that grows with high-dimensional data and limited labels. ALFA-Mix probes labelled–unlabelled feature interpolations for informative prediction inconsistencies.
1. Introduction
- Batch AL fixes the number of instances sent to an oracle each round, and repeated rounds make choosing the most valuable current examples its core challenge.High-quality annotations can be slow and expensive.Source: §1 · p. 1
- Deep AL remains challenging with high-dimensional data and few labels; output-only uncertainty methods can miss feature information, while BADGE’s high-dimensional gradient embeddings are costly.The introduction says BADGE performs relatively poorly in low-data regimes and can be impractical for deep models, large datasets, and many classes.Source: §1 · pp. 1–2
- ALFA-Mix forms convex combinations of labelled and unlabelled latent representations and identifies informative instances through variability in their predicted labels.It explores the neighbourhood around an unlabelled instance by interpolating its features with labelled ones.Source: Figure 1 · p. 1, §1 · p. 2
- Under a norm constraint, optimal interpolation uses feature differences and the model’s feature gradient; class-specific anchors then support candidate selection and centroid-based batch diversity.The paper states that the optimal interpolation has a closed-form solution, enabling deterministic selection at minimal computing cost.Source: §1 · pp. 2–3
- ALFA-Mix outperforms counterparts across 9 image, 2 OpenML, and one video datasets, with particularly significant gains in low-data regimes and 43% better performance than state-of-the-art on a self-trained vision transformer.The paper also reports significantly better results for video classification using transformers.Source: §1 · p. 3
2. Related Work
Prior active-learning work combines uncertainty and diversity estimates or uses separate selection models, but each family has limitations. ALFA-Mix is positioned as an approach that uses latent features while avoiding repeated, costly, or out-of-distribution selections.
2. Related Work
- Diversity methods cluster feature representations or construct latent-feature core-sets [35] to select samples that represent the unlabelled pool.Source: §2 · p. 3
- Uncertainty methods use predicted likelihoods to select boundary-near samples but ignore feature representations; DFAL [13] perturbs inputs, while ALFA-Mix interpolates in latent space and is faster and more accurate.The related-work discussion warns that random perturbations may fall outside the true data distribution and waste labeling on implausible feature values.Source: §2 · p. 3
- Separate selection models add hyperparameter sensitivity and computation, omit sample diversity, risk repetitive selections, and perform poorly in low-data regimes.Source: §2 · p. 3
- Hybrid methods combine informativeness and representativeness, learn to combine strategies, select diverse contexts, or cluster output-layer gradients, which is not scalable to large class counts.The paper says ALFA-Mix consistently outperforms the gradient-clustering method and is efficient and scalable to large tasks.Source: §2 · p. 3
3. Methodology
ALFA-Mix probes unlabeled examples by mixing their representations with labeled class anchors and checking how the model’s predictions and loss change. It selects examples with prediction changes, choosing the mixing ratio through constrained optimization and reducing the candidate set through clustering.
3.1. Problem Definition
- Each round selects B unlabeled examples for oracle labeling in a supervised K-class task, then evaluates the updated learner on an unseen test set.The learner is a backbone-plus-classifier network; class-wise mean labeled representations serve as anchors.Source: §3.1 · pp. 3–4
3.2. Feature Mixing
- ALFA-Mix interpolates each unlabeled representation with class anchors and probes prediction and loss changes to detect features novel to the model.Feature differences indicate what may be novel, while the loss gradient measures the model’s sensitivity to those features.Source: §3.2 · p. 4
3.3. Optimising the Interpolation Parameter α
- The per-instance mixing ratio is chosen to maximize interpolation-induced pseudo-label loss change under a mixing constraint, using a dual-norm approximation for efficiency.The approximation is reported to have no significant detrimental effect on final results compared with directly optimizing the loss.Source: §3.3 · pp. 4–5
3.4. Candidate Selection
- Instances whose predictions change under interpolation form the candidate set; k-means groups them into B clusters, and the nearest-to-center samples are sent for labeling.This clustering reduces a potentially larger candidate set while seeking diversity among the selected samples.Source: §3.4 · p. 5
4. Experiments and Results
Across 30 active-learning settings, ALFA-Mix generally matches or outperforms the strongest baselines, with especially strong results in video classification, low-data settings, and self-supervised vision-transformer settings. Ablations examine its selection and interpolation choices, while acquisition-time results report competitive or faster selection than several baselines.
4.1. Baselines
- The experiments compare ALFA-Mix with random, uncertainty-based, diversity-based, model-based, and hybrid active-learning baselines, including BADGE and CDAL.The comparison spans 30 settings that vary by dataset, initial labelled samples, network, AL rounds, and batch budget.Source: §4.2 · p. 5, §4.1 · p. 5
4.2. Experiment Settings
- Experiments span image and non-image datasets, small and large budgets, and different architectures and initialisations; most settings run for 10 rounds and are repeated five times.DomainNet-Real uses five rounds; the video experiments use HMDB and repeat each experiment twice.Source: §4.2 · pp. 5–6
4.3. Overall Results
- In 282 of 285 AL rounds, ALFA-Mix matched or outperformed the best-performing baselines BADGE and CDAL; it was outperformed in only 3 rounds.Across 30 settings, it outperformed CDAL in 12.3 experiments and BADGE in 10.6 experiments, on average.Source: Figure 4 · p. 6, §4.3 · pp. 6–7
- On HMDB video-classification experiments, ALFA-Mix outperformed all other strong baselines by more than 2% in the first three active-learning rounds and improved Top-1 test accuracy over random sampling by more than 5% in the first two.Table 1 reports Top-1 test accuracy on HMDB.Source: §4.3 · p. 7, Table 1 · p. 7
4.4. Ablation Study
- With self-supervised pre-trained vision transformers, ALFA-Mix beats Random, BALD, CoreSet, and GCNAL in all settings and improves on BADGE and CDAL in 60% and 43% of rounds, respectively.It also beats BADGE in 46% of small-budget experiments and outperforms all methods in at least 60% of the described first-five-round, low-data experiments.Source: §4.4 · p. 7
- Uniform selection from prediction-changed candidates outperforms the tested norm and symmetric-KL heuristics, while k-MEANS outperforms k-MEANS++ in these experiments.The authors suggest the heuristics may spend too much of the budget on samples from a small region; k-MEANS is described as better representing interpolated features.Source: §4.4 · p. 8
- Skipping interpolation-parameter learning reduces the candidate-set size, and anchor-based interpolation outperforms random labelled samples and removing the anchor.The candidate reduction can harm the diversity of samples selected during clustering.Source: §4.4 · p. 8
- For MLP and DenseNet-121, ALFA-Mix is competitive with the fastest baselines and reduces BADGE acquisition time by more than a factor of 2 on datasets with few classes.The authors attribute its speed partly to back-propagating only to a latent representation layer; Table 2 compares label-acquisition runtimes.Source: Table 2 · p. 8, §4.4 · p. 8
5. Conclusions and Limitations
ALFA-Mix achieves state-of-the-art results across varied image, non-image, and video settings, with its largest gains when labelled data and budgets are small. The authors note that latent-space interpolation can produce unexpected representations and that real-world evaluations would benefit the field.
Conclusions
- With small labelled sets and limited budgets, ALFA-Mix surpassed all other baselines in around 60% of evaluated rounds.The authors report its greatest performance boost in this regime.Source: §5 · p. 8
- ALFA-Mix achieved state-of-the-art results across varied image, non-image, and video datasets and settings.The method is based on interpolation between labelled and unlabelled samples.Source: §5 · p. 8
Limitations
- Feature representations are not generally disentangled, so high-dimensional interpolation may yield representations for unexpected inputs.The authors identify real-world evaluation as an area from which the field and their approach could benefit.Source: §5 · pp. 8–9
1. Methodology
The methodology derives an efficient interpolation rule and relates feature mixing to distribution differences. It also contrasts anchor-based mixing with BADGE, adversarial perturbations, and iterative optimization.
1. Methodology
- The 2-norm-constrained interpolation optimization is reduced to a deterministic solution using feature differences and the model's feature gradient.The derivation uses a dual-norm approximation to obtain the optimal interpolation ratio.Source: §1 · p. 11
1.1. Relations Between ALFA-Mix and Other Baselines
- ALFA-Mix tests prediction consistency along anchor-directed feature interpolations, whereas BADGE relies on gradients of unlabelled instances.When predictions are accurate and confident, small movements toward anchors should not change them; inconsistency can indicate a changed surface or novel unlabelled features.Source: §1.1 · p. 11, Figure 7 · p. 12
- Adversarial feature perturbation performs better than BADGE but falls considerably behind ALFA-Mix in the comparison described.The authors attribute ALFA-Mix's advantage to considering both feature novelty and gradient extent, and report that it finds inconsistencies across more of the decision boundary.Source: §1.1 · p. 11
- Interpolation changes the difference between labelled and unlabelled latent-distribution expectations by a constant factor, acting as a surrogate divergence measure.The passage relates this view to methods focused on distributional differences.Source: §1.1 · pp. 11–12
- Five-step iterative gradient optimization has similar performance to the closed-form approach but takes 5× longer.The iterative alternative optimizes the interpolation ratio for each unlabelled instance and anchor.Source: §1.1 · pp. 12–13
2. Experiments
The experiments assess AL methods through repeated pairwise comparisons and examine selected batches for uncertainty and diversity. ALFA-Mix is reported to outperform baselines consistently across datasets, budgets, architectures, and initialization choices.
2.1. Comparison matrix
- The pairwise comparison matrix aggregates round-level wins, using a t-score threshold of 2.776 to decide whether one method beats another.The matrix sums results across settings, each repeated with five random seeds.Source: §2.1 · p. 13
2.2. Sampling Diversity and Uncertainty
- Selected batches had uncertainty closer to the highest possible level than BADGE's and showed diversity in ground-truth labels and decision-boundary positions.The authors conclude that the method selects diverse samples close to decision boundaries without explicitly enforcing boundary proximity.Source: §2.2 · p. 13
2.4. All the Experiments
- The evaluation covered 285 active-learning rounds across 30 settings defined by dataset, budget, model architecture, and initialization.The experiment settings also include different model and dataset configurations.Source: §2.4 · p. 13
2.3. More Ablations
- ALFA-Mix consistently outperformed other baselines across data, budget sizes, model architectures, and network initializations, and was rarely beaten.The pairwise comparisons cover data type, budget size, architecture, and initialization.Source: §2.3 · p. 13, Figure 11 · p. 15, Figure 12 · p. 16