Source-linked AI summary
Learning from Between-class Examples for Deep Sound Recognition
Yuji Tokozume, Yoshitaka Ushiku, Tatsuya Harada
TL;DR
Sound recognition performance depends partly on how training data are fed, motivating a learning method beyond conventional augmentation. BC learning mixes sounds from different classes and trains models to predict the mixing ratio, improving performance across networks, datasets, and augmentation schemes; EnvNet-v2 reaches a 15.1% error rate on ESC-50, surpassing the human level.
Problem
Sound-recognition performance depends on the amount and quality of training data and on how training data are fed.
Method
BC learning mixes two sounds from different classes using a random ratio and trains the model to output that mixing ratio.
Results
BC learning improves performance across various sound-recognition networks, datasets, and data-augmentation schemes, with EnvNet-v2 achieving a 15.1% error rate on ESC-50.
Takeaways & Limitations
BC learning provides a learning method that uses between-class examples to learn a discriminative feature space and improve sound-recognition performance.
Takeaways & Limitations
BC learning requires more training epochs than standard learning to achieve improved performance in the reported EnvNet experiments.
Abstract
from arXiv · showhide
Deep learning methods have achieved high performance in sound recognition tasks. Deciding how to feed the training data is important for further performance improvement. We propose a novel learning method for deep sound recognition: Between-Class learning (BC learning). Our strategy is to learn a discriminative feature space by recognizing the between-class sounds as between-class sounds. We generate between-class sounds by mixing two sounds belonging to different classes with a random ratio. We then input the mixed sound to the model and train the model to output the mixing ratio. The advantages of BC learning are not limited only to the increase in variation of the training data; BC learning leads to an enlargement of Fisher's criterion in the feature space and a regularization of the positional relationship among the feature distributions of the classes. The experimental results show that BC learning improves the performance on various sound recognition networks, datasets, and data augmentation schemes, in which BC learning proves to be always beneficial. Furthermore, we construct a new deep sound recognition network (EnvNet-v2) and train it with BC learning. As a result, we achieved a performance surpasses the human level.
1 INTRODUCTION
BC learning trains sound-recognition models on mixtures of sounds from different classes, learning to predict their mixing ratio rather than a single class label. The method is presented as improving feature-space discrimination and performance across settings, including a 15.1% ESC-50 error rate with EnvNet-v2.
- 1 INTRODUCTION: BC learning mixes two sounds from different classes at a random ratio and trains the model to predict that ratio.The method uses mixed data and labels rather than pure examples during training.
- 1 INTRODUCTION: The method targets a discriminative feature space by treating between-class sounds as between-class examples.The paper distinguishes this learning objective from simply increasing training-data variation.
- 1 INTRODUCTION: BC learning enlarges Fisher’s criterion and regularizes the positional relationship among class feature distributions.Fisher’s criterion is described as the ratio of between-class distance to within-class variance.
- 1 INTRODUCTION: BC learning improved performance across sound-recognition networks, datasets, and data-augmentation schemes.The paper reports that the method was consistently beneficial across these settings.
- 1 INTRODUCTION: 15.1% error rate was achieved on ESC-50 with EnvNet-v2 trained using BC learning, surpassing the human level.EnvNet-v2 is introduced as a new deep sound-recognition network.
2 RELATED WORK
Related work applies deep networks to sound recognition using engineered features or raw waveforms, while data augmentation increases training variation through transformations, cropping, and mixtures. Other approaches incorporate external data or knowledge, such as paired image–sound videos and transferred image-recognition representations.
- 2 RELATED WORK: Logmel-CNN classifies two-channel feature maps formed from static and delta log-mel features with a 2-D CNN.The feature map arranges log-mel frames along time and exploits locality in time and frequency.
- 2 RELATED WORK: SoundNet and other networks learn sound representations directly from 1-D raw waveforms using convolutional and pooling layers.SoundNet uses unlabeled videos, while Dai et al. report their best result with an 18-layer network.
- 2 RELATED WORK: Cropping, time stretching, pitch shifting, and background-noise addition increase training-data variation for sound recognition.Cropping also supports efficient network training by using short sections of training sounds.
- 2 RELATED WORK: External-data approaches learn sound representations from paired image–sound videos or transfer knowledge from pretrained image-recognition networks.Aytar et al. minimize KL divergence between image-network and sound-network output predictions.
3 BETWEEN-CLASS LEARNING FOR SOUND RECOGNITION
BC learning mixes sounds from different classes and trains the model to predict their mixing ratio, shaping a more discriminative and regularized feature space. Its mixing procedure accounts for sound energy and auditory perception, while optimization uses KL-divergence on generated mini-batches.
- BC learning pipeline: BC learning mixes two sounds from different classes with a random ratio and trains the model to output that ratio.Mixed labels are formed using the same ratio applied to the two one-hot labels.
- Mixing method: The proposed sound mixture accounts for sound energy by normalizing the weighted waveform combination.The paper states that this method performs better than the simpler weighted sum in experiments.
- Mixing method: A-weighted sound pressure levels are used to calibrate mixing coefficients toward the intended auditory ratio, with G defined as the maximum short-window level.The authors report that replacing A-weighting with RMS energy worsens performance.
- Enlargement of Fisher’s criterion: BC learning enlarges Fisher’s criterion by penalizing mixed sounds that project into either original class instead of between-class locations.Fisher’s criterion is the ratio of between-class distance to within-class variance.
- Enlargement of Fisher’s criterion: PCA visualization shows mixed-sound features distributed between the two source classes.
- Regularization of positional relationship among feature distributions: BC learning regularizes class positions by discouraging mixed sounds from crossing another class’s decision boundary.Standard learning can classify dog-bark/rain mixtures as baby cry for mixing ratios from 0.45–0.8.
4 EXPERIMENTS
Experiments across networks, datasets, augmentation schemes, and ablations show that BC learning consistently improves sound recognition, while its benefits depend on sufficient training and specific mixing choices.
- Comparison between standard learning and BC learning: BC learning improved all evaluated networks on all datasets, with gains of 4.5–6.4% on ESC-50, 1.5–4.0% on ESC-10, and 1.8–4.8% on UrbanSound8K.The comparison used EnvNet, SoundNet5, M18, and Logmel-CNN + BN.
- Experiment on a deeper network: EnvNet-v2 achieved larger BC-learning improvements of 7.4%, 3.6%, and 7.5% on ESC-50, ESC-10, and UrbanSound8K, respectively.EnvNet-v2 uses a deeper architecture and a 44.1 kHz input sampling rate than EnvNet.
- Experiment with strong data augmentation: 15.1% error on ESC-50 with EnvNet-v2 and strong augmentation surpassed the reported human performance of 18.7%.BC learning remained beneficial when scale and gain augmentation were added.
- Relationship with # of training epochs: BC learning required more training epochs: performance improved at 900 epochs on ESC-10 and 1,200 epochs on ESC-50, whereas standard learning was sufficient at approximately 600 epochs.With too few epochs, BC learning performed worse than standard learning; the required epoch count may increase with more classes.
- Ablation analysis: The proposed sound-pressure-aware mixing method and ratio label performed best, indicating that mixing design and target construction affect BC-learning performance.The ratio label outperformed dominant-sound and multi-label alternatives, while Eqn. 2 with A-weighting was the best mixing method.
- Ablation analysis: Mixing two sounds in the input space performed best, while mixing three sounds performed worse than mixing two despite greater training-data variation.Input-space mixing also avoids additional forward/backward computation and is easy to implement.
5 CONCLUSION
The paper presents BC learning as a simple method that improves deep sound recognition across varied settings and supports learning discriminative feature spaces from between-class examples.
- BC learning improves performance across various sound recognition networks, datasets, and data augmentation schemes.
- Training EnvNet-v2 with BC learning achieves performance surpassing the human level.
- BC learning learns a discriminative feature space from between-class examples without inputting pure examples.
- The authors argue that BC learning’s core idea could contribute to improving performance in tasks involving other modalities.
A LEARNING SETTINGS
The learning settings specify scheduled learning-rate decay, warmup, training duration, and longer schedules for BC learning.
- Standard learning begins with Initial LR, uses a 0.1× smaller rate during Warmup epochs, and divides the rate by 10 at scheduled epochs.
- Training terminates after the configured number of epochs, while BC learning doubles both the training duration and learning-rate schedule.
- Table 3 presents the detailed learning settings used for standard learning.
B CONFIGURATION OF ENVNET-V2
EnvNet-v2 is a deeper, high-resolution raw-waveform network with 10 convolutional layers, 3 fully connected layers, and 5 max-pooling layers.
- EnvNet-v2 consists of 10 convolutional layers, 3 fully connected layers, and 5 max-pooling layers.
- The network uses a 44.1 kHz sampling rate to retain rich high-frequency information.
- Table 4 reports EnvNet-v2’s configuration, with data shape expressed as channel, frequency, and time dimensions.