PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Sound Event Classifiers from Web Audio with Noisy Labels

Eduardo Fonseca, Manoj Plakal, Daniel P. W. Ellis, Frederic Font, Xavier Favory, Xavier Serra

arXiv:1901.01189v2cs.SDcs.LGeess.ASstat.ML

TL;DR

Large-scale sound-event datasets trade label quality for scale, while metadata-derived web labels can be noisy and their effects are little studied in sound classification. This paper introduces FSDnoisy18k and evaluates CNN training with noisy data and noise-robust losses, finding noisy-data training can outperform smaller clean-data training and some losses improve performance.

  • Problem

    Manual labeling limits dataset scale, while web labels inferred from metadata can be noisy; the impact of label noise has received little attention in sound event classification.

    Source: §1 · p. 1

  • Method

    The paper presents FSDnoisy18k, a dataset with a small clean portion and larger noisy portion, characterizes its label noise, and evaluates a CNN baseline with noise-robust losses.

    Source: §1 · p. 1, §2.2 · p. 3, §5 · p. 4

  • Results

    Noisy-subset training boosts accuracy 6.3% over clean-subset training, while Lq raises baseline accuracy 2.7% on the entire train set and 1.9% on the noisy subset.

    Source: §5.1 · p. 4, §5.2 · p. 4

  • Takeaways & Limitations

    FSDnoisy18k is an expandable resource for investigating label noise and approaches such as semi-supervised learning and learning with minimal supervision.

    Source: §2.2 · p. 3

Abstract

from arXiv · show

As sound event classification moves towards larger datasets, issues of label noise become inevitable. Web sites can supply large volumes of user-contributed audio and metadata, but inferring labels from this metadata introduces errors due to unreliable inputs, and limitations in the mapping. There is, however, little research into the impact of these errors. To foster the investigation of label noise in sound event classification we present FSDnoisy18k, a dataset containing 42.5 hours of audio across 20 sound classes, including a small amount of manually-labeled data and a larger quantity of real-world noisy data. We characterize the label noise empirically, and provide a CNN baseline system. Experiments suggest that training with large amounts of noisy data can outperform training with smaller amounts of carefully-labeled data. We also show that noise-robust loss functions can be effective in improving performance in presence of corrupted labels.

1. INTRODUCTION

Manual labeling limits audio dataset size and forces a trade-off between scale and label quality, while web metadata can enable rapid collection at the cost of label noise. The paper introduces FSDnoisy18k for studying real label noise and evaluates noise-robust losses for sound classification.

  • Manual labeling is costly and often limits audio dataset size, forcing a trade-off between scale and label quality.Source: §1 · p. 1
  • Inferring labels from user metadata enables rapid web-audio collection but risks substantial noise from unreliable metadata and label mapping.Source: §1 · p. 1
  • FSDnoisy18k provides singly labeled audio with a small clean portion and a much larger real-world noisy portion, plus empirical noise characterization and a CNN baseline.Source: §1 · p. 1
  • Label-noise research has received little attention in sound event classification, and prior audio work had not specifically supplied a dataset for studying it.Source: §1 · p. 1
  • The paper’s second contribution is an empirical evaluation of noise-robust loss functions, some used for the first time in sound classification.Source: §1 · p. 1

2. DATASET

FSDnoisy18k combines Freesound-derived clips with tag-based candidate labels and a smaller verified clean subset. Its noisy data contains varied label errors, and the dataset supports study of real-world noise and other learning settings.

2. DATASET

  • FSDnoisy18k uses Freesound tags to generate weak clip-level candidate labels, with a portion human-validated and remaining unvalidated tag-only clips designated noisy.The clean portion is based on clips verified as Present and Predominant; multiple labels were reduced to one.Source: §2 · pp. 1–2

2.1. Label Noise Characteristics

  • The taxonomy separates incorrect from incomplete labels and classifies the true or missing label as in-vocabulary or out-of-vocabulary.Examples include Wind clips labeled as Rain (incorrect/IV) and clips containing both Slam and Squeak with only one label (incomplete/IV).Source: §2.1 · p. 2
  • Web-audio-specific noise also includes ambiguous labels and false strong labeling, where patches inherit clip labels even when the labeled sound is inactive.Ambiguity can arise with human imitations or heavily processed sounds.Source: §2.1 · p. 2
  • 60% of the analyzed noisy-data labels show noise, most often incorrect or incomplete OOV labels; per-class total noise ranges roughly from 20% to 80%.About 40% are correct and complete; the distribution is reported for a random 15% sample of noisy data.Source: §2.1 · pp. 2–3, Table 1 · p. 2

2.2. Dataset Characteristics

  • The dataset has 18,532 clips (42.5 hours) across 20 classes; its clean test set contains 947 clips, while the training set is about 90% noisy by clip count.The dataset is expandable, with per-class variation in noise types and amounts, and supports semi-supervised and minimal-supervision investigations.Source: §2.2 · p. 3

3. BASELINE SYSTEM

The baseline converts clips into 2-second log-mel patches and classifies them with a compact CNN. Its stated purpose is to establish a reference accuracy, not to maximize performance.

3. BASELINE SYSTEM

  • Variable-length clips are represented as 2-second, 96-band log-mel patches; short clips are replicated and longer clips trimmed into patches carrying the clip label.Source: §3 · p. 3
  • The baseline uses a three-convolution, one-dense-layer CNN with batch normalization and pre-activation, totaling approximately 0.5M weights.Clip predictions aggregate patch-level predictions with a geometric mean.Source: §3 · p. 3
  • The system is intended to provide a reference accuracy rather than maximize performance; extensive hyperparameter tuning and model exploration were not conducted.Source: §3 · p. 3

4. NOISE-ROBUST LOSS FUNCTIONS

The section describes three ways to make classification losses more robust to noisy labels: update targets using model predictions, balance CCE and MAE behavior, or discard suspiciously high losses.

4. NOISE-ROBUST LOSS FUNCTIONS

  • CCE measures divergence between predicted class probabilities and one-hot target labels, and corrupted targets can make weight updates suboptimal.The paper evaluates the robust losses as modifications of CCE.Source: §4 · p. 3
  • Lsoft replaces the target with a convex combination of the noisy label and the model’s current prediction, shifting attention toward predictions as training progresses.This soft bootstrapping approach is intended to give less weight to noisy labels.Source: §4 · p. 3
  • Lq generalizes CCE and MAE to balance CCE’s stronger weighting of mismatched predictions with MAE’s equal weighting, which is theoretically robust to corrupted labels.Preliminary experiments with MAE alone performed poorly on FSDnoisy18k.Source: §4 · pp. 3–4
  • Loss masking first computes CCE per example, then excludes losses above a threshold from the minibatch total because they may indicate corrupted labels.The tested thresholds are a fraction of the maximum loss or the median plus a multiple of the standard deviation.Source: §4 · p. 4

5. EXPERIMENTS AND DISCUSSION

Experiments compare training subsets and noise-robust losses under the baseline learning setup. Noisy data can outperform clean-only training, and Lq improves accuracy over the baseline on both the full and noisy subsets.

5.2. Noise-Robust Loss Functions

  • 2.7% and 1.9%: Lq raised accuracy over the baseline on the entire train set and noisy subset, respectively.Lq was the top-performing loss on both subsets, followed by Lsoft and then the heuristic approaches.Source: Table 2 · p. 4, §5.2 · p. 4

5.1. Baseline System

  • 6.3%: noisy-subset training exceeded clean-subset accuracy despite using roughly an order of magnitude more data; adding clean data to noisy training raised accuracy 5.1%.The authors note the noisy-data advantage comes with substantially more data, while clean curation requires significant effort.Source: Table 2 · p. 4, §5.1 · p. 4

5.2. Noise-Robust Loss Functions

  • Lq works with both out-of-vocabulary and in-vocabulary noisy labels, the two noise types present in FSDnoisy18k.Source: §5.2 · p. 4
  • Applying robust losses only to noisy-subset examples led to slightly better performance than applying them indiscriminately to all training data.The clean subset continued to use regular CCE.Source: §5.2 · p. 4

6. CONCLUSION

FSDnoisy18k supports investigation of real label noise in sound event classification. Its conclusion characterizes the noise and reports that noisy web audio and some noise-robust losses can be useful for training sound event recognizers.

6. CONCLUSION

  • Some evaluated noise-robust loss functions significantly improve performance with corrupted labels while requiring minimal engineering effort.Source: §6 · p. 4
  • ≈60% of the noisy data has label noise, most of it out-of-vocabulary noisy labels.Source: §6 · p. 4
  • Large amounts of Freesound audio supervised by user-generated tags can be a feasible option for training sound event recognizers.Source: §6 · p. 4
  • FSDnoisy18k combines a small amount of manually labeled data with a large amount of noisy data and varying per-class noise types and amounts.Source: §6 · p. 4
Loading 1901.01189v2…