Source-linked AI summary
The Benefit Of Temporally-Strong Labels In Audio Event Classification
Shawn Hershey, Daniel P W Ellis, Eduardo Fonseca, Aren Jansen, Caroline Liu, R Channing Moore, Manoj Plakal
TL;DR
AudioSet commonly uses temporally weak labels, leaving the value of precise event timing uncertain. This paper adds strong labels and explicit-negative evaluation data to a subset of AudioSet, then tests mixed weak-and-strong training. Fine-tuning with strong labels improves performance on both strong and original weak evaluation, and the released labels extend AudioSet’s annotation resources.
Problem
Audio event classification relies on large datasets whose 10-second weak labels provide limited temporal precision, raising uncertainty about how much accuracy is impaired by that weakness.
Method
The paper collects approximately 0.1-second strong labels, constructs strong evaluation data with explicit negatives, and fine-tunes models using strong labels alongside the original weakly labeled AudioSet data.
Results
Fine-tuning with strong labels substantially improves classifier performance, including on the original weak evaluation, with the strongest reported strong-evaluation d' reaching 1.41.
Takeaways & Limitations
Using fine-time-resolution labels for even a few percent of training data can improve classifiers pretrained on a large weakly labeled dataset.
Abstract
from arXiv · showhide
To reveal the importance of temporal precision in ground truth audio event labels, we collected precise (~0.1 sec resolution) "strong" labels for a portion of the AudioSet dataset. We devised a temporally strong evaluation set (including explicit negatives of varying difficulty) and a small strong-labeled training subset of 67k clips (compared to the original dataset's 1.8M clips labeled at 10 sec resolution). We show that fine-tuning with a mix of weak and strongly labeled data can substantially improve classifier performance, even when evaluated using only the original weak labels. For a ResNet50 architecture, d' on the strong evaluation data including explicit negatives improves from 1.13 to 1.41. The new labels are available as an update to AudioSet.
1. INTRODUCTION
Audio event classifiers depend on large datasets, but those datasets usually use temporally weak labels because precise annotation is expensive. The paper tests whether adding a small set of temporally strong labels can improve performance alongside abundant weakly labeled data.
- 1. INTRODUCTION: Large-scale sound event classifiers generally rely on weak labels that indicate whether an event occurs within a 10-second clip without precise timing.These annotations are faster to collect than temporally precise labels.
- 1. INTRODUCTION: The strong-label training subset covers about 4% of the 1.8M weakly labeled training clips.The paper therefore considers whether label precision can offset the scarcity of strongly labeled data when used alone or in combination.
- 1. INTRODUCTION: Combining the large weakly labeled dataset with a small strongly labeled subset is proposed as a way to outperform training on either dataset individually.This is the central experimental motivation of the paper.
2. RELATED WORK
Prior work provides strong labels mainly in small real-world datasets or synthetic soundscapes, while large-scale AudioSet systems have largely relied on weak supervision. This work addresses the limited study of combining strong and weak labels by adding substantial strong labeling to AudioSet.
- 2. RELATED WORK: Strongly labeled real-world sound event datasets have typically been small because manually marking event boundaries is time-consuming.Examples cited include roughly 2 hours for each TUT Sound Events dataset and less than 5 hours of strong labels in a DCASE AudioSet task.
- 2. RELATED WORK: Synthetic soundscapes provide precise event times and potentially unlimited training data, but may not represent real-world acoustic conditions.Their advantages also include control over event amplitudes and dataset composition.
- 2. RELATED WORK: Weak-label research has improved AudioSet recognition through class-imbalance mitigation, augmentation, pooling, and attention, but evaluation has been difficult without strongly labeled tests.Some methods still produce scores at fine time resolutions despite weak training labels.
- 2. RELATED WORK: Research exploiting both strong and weak labels remains scarce, although prior work reported gains from adding a small amount of strongly labeled data.The paper investigates this combination in the AudioSet setting.
- 2. RELATED WORK: The paper contributes over 200 hours of strong positive and explicit negative labels across 356 AudioSet classes.This increases the scale of manually strong-labeled real-world audio available for studying mixed supervision.
3. STRONG-LABELED DATASET
The paper constructs a strong-labeled AudioSet subset and evaluation set by marking precise event extents, explicit negatives, and complementary negative frames. The release covers 356 classes, while analysis shows changed class priors and several remaining scope and labeling limitations.
- 3. STRONG-LABELED DATASET: The original AudioSet labels are limited by 10-second temporal precision, missing positives, and occasional use of overly abstract parent labels.These limitations arise because annotators confirmed only selected labels and were not asked to mark every event.
- 3. STRONG-LABELED DATASET: Annotators marked every perceived event on spectrogram timelines, assigned the most-specific searchable label, and achieved timing judged precise to at least 0.1 seconds.The collection process was designed to address temporal imprecision, missing positives, and label abstraction.
- 3. STRONG-LABELED DATASET: The selected data include 18k surviving evaluation clips and about 67k training clips sampled with an attempted target of 250 clips per class.Class co-occurrence caused many labels to receive more than the target number of selected clips.
- 3. STRONG-LABELED DATASET: Musical instrument and music genre labels were merged into a single “Music” label, eliminating 140 of the original 527 categories.The paper states that instrument-level annotation in ensemble recordings was especially arduous.
- 3. STRONG-LABELED DATASET: Each labeling was produced by a first annotator and one reviewing annotator in a pipelined process, without estimated inter-annotator agreement.Labels were accepted after review when no changes were made, and quality was assessed through spot-checking.
- 3. STRONG-LABELED DATASET: Strong labels were projected onto 960 ms frames when a frame was at least 50% occupied by an event or contained at least 50% of the event’s total duration.The second condition preserves strong segments shorter than 480 ms.
- 3. STRONG-LABELED DATASET: The evaluation uses explicit negatives from plausible candidate clips and complementary negative frames surrounding positive event regions.This design tests both class discrimination and contrast between event-containing regions and nearby context.
- 3. STRONG-LABELED DATASET: Strong positives and explicit negatives are released for 356 of the 527 AudioSet classes, with full original-time-resolution labels for training and evaluation clips.The released training subset contains 66,924 clips.
4. EXPERIMENTS
Experiments compare weak and strong labeling regimes, including mixtures used to fine-tune a model pretrained on 1.8M weakly labeled clips. Strong labels produce the best Strong-evaluation results, with gains in d′ attributable partly to temporal precision and increasing with more strong data.
- Training sets: The experiments compare Weak-1.8M, Strong-67k, Weak-67k, Diffuse-67k, and mixtures used for fine-tuning.Diffuse-67k expands strong labels across each full 10-second clip, while mixtures are controlled by μ.
- Results: Strong-67k produces the best results among the 67k-clip training sets on the Strong evaluation.Strong labels improve Strong-evaluation d′ by 0.17 and Weak-evaluation d′ by 0.14 relative to the corresponding comparison.
- Results: Pretraining on Weak-1.8M followed by fine-tuning on a Weak-1.8M and Strong-67k mixture gives the best results.Training on Weak-1.8M alone substantially outperforms training on Strong-67k alone.
- Results: Fine-tuning with Strong rather than Diffuse labels improves Strong-evaluation d′ by 0.11.The authors suggest the 0.26 d′ gain from Weak-1.8M to +Strong comprises 0.11 from temporal precision and 0.15 from other factors.
- Strong-data proportion: Both metrics improve steadily as the Strong-data proportion rises from 10% to 100%, with diminishing returns at 67k clips.The figure evaluates d′ and lωlrap on the Strong evaluation data.
5. CONCLUSIONS
The paper finds that fine-time-resolution strong labels for even a few percent of training data can improve a classifier fine-tuned from a large weakly labeled dataset. It also identifies future directions involving mixed-quality labels and direct segment-boundary prediction.
- Conclusion: Strong labels for even a few percent of training data deliver significant improvements when fine-tuning a classifier pretrained on weak labels.The evaluations use fixed-size frames, while strong labels suggest future approaches that directly predict segment boundaries.