Source-linked AI summary
Geometric Consistency Based Label-Free Validation for Checkpoint Selection in Audio Active Learning
Shiqi Zhang, Tuomas Virtanen
TL;DR
Audio active learning checkpoint selection commonly relies on validation labels, which consume part of a limited annotation budget. Geometric consistency uses pool pseudo-label agreement and labeled-set fit to select checkpoints; across evaluated settings, it ranks first among deployable methods and performs on par with an out-of-budget validation set.
Problem
Validation labels consume annotation budget in audio active learning, while held-out splits shrink training data and refit approaches can reuse selection labels or add training cost.
Source: §1 · p. 1Method
Results
45 of 48 settings: Geo-Con ranks first among six deployable methods and performs on par with an out-of-budget validation set.
Source: §1 · p. 2Takeaways & Limitations
The results suggest the unlabeled pool can provide a reliable model-selection signal, and audio AL evaluations should count validation labels toward the annotation budget.
Source: §6 · p. 4The preliminary study uses frozen embeddings and an MLP head; extension to end-to-end fine-tuning remains future work.
Source: §5 · p. 4
Abstract
from arXiv · showhide
Audio active learning (AL) trains a classifier at low annotation cost by labeling only an informative fraction of an unlabeled collection over successive rounds. Each round picks one of the checkpoints saved during training. Mainstream AL protocols do so with an assumed validation set, but in practice its labels also count toward the annotation cost, shrinking the advantage of an already small annotation budget. We propose geometric consistency, a label-free validation rule for checkpoint selection. Both classifiers, nearest-class-mean and k-nearest-neighbor, are built from the labeled set in a frozen pretrained embedding space. They then predict the unlabeled data to model its geometry. The rule scores each checkpoint by its agreement with these pseudo-labels and its cross-entropy on the labeled set. It works with any query strategy, has one hyperparameter, insensitive in practice, and adds only one batched forward pass over the unlabeled data per round. At matched label budgets across datasets, encoders, and query strategies, it attains the best mean rank in area under the labeling curve (AULC) among methods without extra labels and is on par with an out-of-budget validation set. To our knowledge, this is the first systematic study of label-free validation for checkpoint selection in audio AL.
1. INTRODUCTION
Audio active learning must choose checkpoints without letting validation labels silently consume a scarce annotation budget. Geometric consistency uses pool geometry and labeled-set fit for selection, and performs strongly under unified label accounting.
1. INTRODUCTION
- Validation labels count toward the annotation budget, so held-out validation shrinks training data while refit methods reuse selection labels and can add training cost.K-fold refit further multiplies per-round training cost, and small validation splits may poorly represent the data distribution.Source: §1 · p. 1
- Existing label-free criteria monitor training dynamics or decide when to stop labeling, while self-generated pool pseudolabels were rejected as biased.The unlabeled pool is already used by acquisition strategies, making it a potential source of selection information without extra annotation.Source: §1 · p. 1
- Geometric consistency scores checkpoint predictions against NCM and k-NN pseudo-labels from labeled-set embeddings, alongside labeled-set cross-entropy.The pseudo-labelers are independent of the candidate head under evaluation.Source: §1 · pp. 1–2
- Geo-Con ranks first in 45 of 48 settings among deployable methods and performs on par with an out-of-budget validation set.It attains mean rank 1.08 among six deployable methods and a mean 2.8 pp AULC gain over holdout validation.Source: §1 · p. 2
2. PROBLEM SETUP
The paper studies checkpoint selection in pool-based audio active learning, where each round adds queried labels and trains a shallow head over frozen-encoder embeddings. The selected checkpoint also determines the model used for the next round’s acquisition.
2. PROBLEM SETUP
- Pool-based active learning begins with an unlabeled audio set and divides it into labeled and unlabeled subsets as rounds progress.A frozen encoder maps clips to embeddings, and a shallow classification head is trained from scratch each round.Source: §2 · p. 2
- Each round queries n_q clips, with cumulative budget m_t = (t+1)n_q, and saves E intermediate head checkpoints.The stopping method chooses one saved checkpoint as that round’s model.Source: §2 · p. 2
- Checkpoint selection uses no labels beyond the current labeled set, and the selected model generates the next round’s acquisition trajectory.Each stopping method therefore produces its own labeling trajectory.Source: §2 · p. 2
3. GEOMETRIC CONSISTENCY
Geometric consistency combines two pool pseudo-label agreement signals with labeled-set fit to select a checkpoint. Its complementary pseudo-labelers capture global class prototypes and local neighborhoods in the frozen embedding space.
3. GEOMETRIC CONSISTENCY
- The two geometric views are intended to reduce agreement when a checkpoint is underfit or overfit, while labeled-set cross-entropy measures fit.The frozen embedding geometry supplies structure before task labels are seen; the two pseudo-labelers provide global and local views.Source: §3 · p. 2
- NCM assigns each pool clip the class whose prototype, the mean of unit-normalized labeled embeddings, has greatest cosine similarity.This global prototype view can be built from very few labels per class and works from early rounds.Source: §3 · p. 2
- The k-NN labeler predicts from the k most similar labeled samples, weighting each vote by cosine similarity and ignoring negative similarities.Its local-neighborhood view complements prototypes when within-class structure is rich and assumes no unimodal clusters.Source: §3 · pp. 2–3
- The method combines pool agreement with NCM and k-NN pseudo-labels and mean cross-entropy on labeled data, then selects the checkpoint with the largest equally weighted standardized score sum.The agreement is macro accuracy over classes present in the pseudo-labels, so rare and common pseudo-label classes weigh equally.Source: §3 · p. 3
- The rule’s only hyperparameter is neighbor count k = 10; it is acquisition-agnostic and adds one batched forward pass of all E checkpoints over cached pool embeddings.Source: §3 · p. 3
4. EXPERIMENTAL SETUP
The evaluation compares checkpoint-selection methods under a unified label budget across audio datasets, frozen encoders, acquisition strategies, and random seeds. It uses aligned checkpoint grids and reports test macro accuracy and AULC.
4. EXPERIMENTAL SETUP
- The study covers four audio datasets, two frozen encoders, six acquisition strategies, eight stopping methods, and ten seeds, for 3,840 runs.The datasets span environmental sound, Freesound content, bird vocalisations, and urban noise; the encoders are CLAP and PANNs CNN14.Source: §4 · p. 3
- Runs last 20 rounds with 50 queried labels per round, up to 1,000 labels, and save 100 checkpoints on a shared step grid.The checkpoint grid is aligned across methods at every budget point.Source: §4 · p. 3
- Baselines include training-dynamics methods and holdout or refit validation methods, with holdout methods reserving 10 of each round’s 50 labels.K-fold refit uses a stratified five-fold split and six trainings per round; holdout-v-refit retrains on all labels after selection.Source: §4 · p. 3
- Evaluation measures test macro accuracy and macro AULC, while unified-budget accounting includes validation labels at every budget point.Selection-signal costs take the form of held-out labels, extra head retraining, or one batched forward pass over the unlabeled pool.Source: §4 · p. 3
5. RESULTS
Geo-Con leads the deployable methods overall and across dataset–encoder blocks, while remaining close to out-of-budget validation. Its performance is consistent across acquisition strategies, and ablations show contributions from both labeled-set fit and pool geometry.
Main results
- Geo-Con reaches 71.99% AULC, 1.01 pp ahead of kfold-refit and 2.80 pp ahead of holdout-v; it is on par with val-best at 72.01%.It ranks first in 45 of 48 settings, while GWA and LabelWave trail by roughly 2 pp AULC; holdout-v-refit trails holdout-v by 0.91 pp.Source: §5 · pp. 3–4
- On near-saturated ESC-50 with CLAP, final-accuracy differences compress while AULC still separates convergence speed.Source: §5 · p. 4
Across datasets and encoders
- Geo-Con is the best deployable method in all eight dataset–encoder blocks; its AULC gain over holdout-v ranges from +1.46 pp to +4.28 pp.The largest gain is on DB3V (+4.28 pp), and the smallest is on UrbanSound8K (+1.46 pp).Source: §5 · p. 4, Table 2 · p. 4
Compatibility with acquisition strategies
- Geo-Con’s mean AULC rank stays at or below 1.25 under all six acquisition strategies, including plain random sampling.Its advantage therefore does not hinge on a particular query strategy.Source: §5 · p. 4, Table 3 · p. 4
Ablations
- Removing the labeled-set fit term costs 0.77 pp AULC, versus 0.17 pp for NCM and 0.07 pp for k-NN; the geometric terms together add 0.25 pp.The NCM term carries most of the geometric gain, and all seven variants retain the best mean rank against baselines.Source: §5 · p. 4, Table 4 · p. 4
- Varying k across a fourfold range changes AULC by at most 0.07 pp, while every tested value keeps the first deployable rank and stays on par with val-best.This preliminary study uses frozen embeddings with an MLP head; end-to-end investigation is left for future work.Source: §5 · p. 4
6. CONCLUSION
Geometric consistency uses the unlabeled pool’s geometry for checkpoint selection without extra labels or retraining, and performs strongly across the reported runs. The conclusion identifies extensions and evaluation-budget practices for future work.
- Across 3840 runs, geometric consistency ranks first among deployable methods and performs on par with an out-of-budget validation set.Source: §6 · p. 4
- The rule draws its checkpoint-selection signal from the unlabeled pool’s geometry at no cost in labels or retraining.Source: §6 · p. 4
- Future work could extend the rule to end-to-end fine-tuning and multi-label tagging, while future audio AL evaluations should count validation labels toward the annotation budget.Source: §6 · p. 4
7. COMPLIANCE WITH ETHICAL STANDARDS
The study analyzes existing public datasets and reports that it required no ethical approval and involved no new human or animal experiments.