Source-linked AI summary
Deep Convolutional Neural Networks and Data Augmentation for Acoustic Event Detection
Naoya Takahashi, Michael Gygli, Beat Pfister, Luc Van Gool
TL;DR
AED methods can lose temporal order by aggregating framewise descriptors, while earlier neural approaches used shallow networks with short input windows. This paper introduces deeper, large-input-field CNNs and data augmentation for event modeling, and reports that the CNN outperforms BoAW and classical DNN methods, with augmentation improving performance by more than 12%.
Problem
AED approaches may lose temporal order through framewise aggregation, while earlier neural systems used shallow, few-frame inputs and relied on HMMs or averaging to model full events.
Source: §1 · p. 1Method
A deeper VGG-inspired CNN uses a large input field to model whole acoustic events end-to-end, alongside data augmentation that introduces variation.
Source: §1 · p. 1, §2.3 · p. 2Results
The deeper CNN significantly outperforms BoAW and classical DNN methods, and the proposed data augmentation improves performance by more than 12%.
Source: §1 · p. 1Takeaways & Limitations
The experiments validate deeper architectures, large input fields, and data augmentation for AED, with augmentation supporting superior performance even when training data is fairly limited.
Source: §4 · p. 4For MIL, the assumption that a positive bag contains at least one positive instance becomes realistic as the number of instances grows, because the probability that all are negative decreases exponentially.
Source: §2.4 · p. 3
Abstract
from arXiv · showhide
We propose a novel method for Acoustic Event Detection (AED). In contrast to speech, sounds coming from acoustic events may be produced by a wide variety of sources. Furthermore, distinguishing them often requires analyzing an extended time period due to the lack of a clear sub-word unit. In order to incorporate the long-time frequency structure for AED, we introduce a convolutional neural network (CNN) with a large input field. In contrast to previous works, this enables to train audio event detection end-to-end. Our architecture is inspired by the success of VGGNet and uses small, 3x3 convolutions, but more depth than previous methods in AED. In order to prevent over-fitting and to take full advantage of the modeling capabilities of our network, we further propose a novel data augmentation method to introduce data variation. Experimental results show that our CNN significantly outperforms state of the art methods including Bag of Audio Words (BoAW) and classical CNNs, achieving a 16% absolute improvement.
1. Introduction
The introduction motivates AED as useful across contextual and multimedia tasks, but notes that existing approaches can lose event-time order or model only short input windows. It presents a deeper, VGG-inspired CNN with a large input field and data augmentation, reporting better performance than prior baselines.
1. Introduction
- AED is relevant to surveillance, multimedia retrieval, and audio segmentation, while non-speech sounds can also provide contextual information for other tasks.Source: §1 · p. 1
- Earlier AED systems used frame-level descriptors that lose temporal order or shallow, few-frame networks whose outputs are aggregated by HMMs or averaging.Source: §1 · p. 1
- The authors introduce CNN architectures with up to nine layers and a large input field to model entire acoustic events directly and train end-to-end.Source: §1 · p. 1
- The deeper CNN significantly outperforms state-of-the-art BoAW and classical DNN methods, while the proposed data augmentation improves performance by more than 12%.Source: §1 · p. 1
- The experiments use a newly created dataset harvested from Freesound for acoustic-event classification.Source: §1 · p. 1
2. Architectural and Training Novelties
The section adapts CNNs to model entire acoustic events and introduces equalized-mixture augmentation to add training variation. It also describes a multiple-instance approach for handling noisy web data.
2.1. Convolutional Network Architecture
- Architecture B uses 9 weight layers, including 6 convolutional and 3 fully connected layers, with 3×3 kernels and ReLU hidden layers.Architecture A has 4 convolutional and 3 fully connected layers; training minimizes cross-entropy with l1 regularization.Source: §2.1 · p. 2
2.2. Large input field
- Unlike short-frame pipelines, the network takes a single multiple-second input to model an entire acoustic event and optimize parameters end-to-end.Source: §2.2 · p. 2
2.3. Data Augmentation
- Equalized Mixture Data Augmentation (EMDA) creates synthetic training sounds by mixing two same-class sources at random timings and modifying their frequency characteristics.The equalizing function is a second-order parametric equalizer; its center frequency, gain, and Q-factor are randomly selected.Source: §2.3 · p. 2
2.4. Multiple Instance Learning
- For noisy web data, MIL uses bags with bag-level labels and shared-weight CNNs, replacing the softmax layer with an aggregation layer.The investigated aggregation functions are max and Noisy OR.Source: §2.4 · pp. 2–3
- Although outliers make it uncertain that a bag contains a positive instance, the authors argue that this assumption becomes realistic as bag size grows.They state that the probability all instances are negative decreases exponentially with the number of instances.Source: §2.4 · p. 3
3. Experiments
The experiments evaluate the dataset and training setup, compare deeper CNNs with baselines, and test input-field size, augmentation, and multiple instance learning. The proposed CNNs outperform previous methods, while the experiments show gains from longer input fields and combined augmentation but not from MIL overall.
3.1. Dataset
- The Freesound-harvested database contains 28 events and is challenging because free-form tags yield irrelevant sounds, while recordings vary by device and conditions, including noisy audio and brief events amid long silences.Sounds longer than 12 seconds were split into pieces shorter than 12 seconds, and samples were converted to 16 kHz, 16-bit mono audio. Only the test set was manually checked, and irrelevant sounds without the target event were omitted.Source: §3.1 · p. 3
3.2. Implementation details
- Experiments used 4-second input patches, balanced augmentation samples by class, and applied dropout to fully connected layers.Testing used 4-second patches with 50% overlap; training randomly cropped 4 seconds per sample.Source: §3.2 · p. 3
3.3.1. State-of-the-art comparison
- 12.5% improvement was achieved for architecture B with data augmentation, and the best result came from B with augmentation.The proposed CNN architectures also outperformed previous methods without augmentation; B outperformed classical DNN/CNN despite having fewer parameters.Source: Table 3 · p. 4, §3.3.1 · p. 4
3.3.2. Effectiveness of large input field
- Larger input fields improved performance, while patches shorter than 1 second showed a sharp drop.The B architecture was tested with patch lengths from 50 to 400 frames, and Figure 3 compares performance against a 30-frame CNN+HMM baseline.Source: Figure 3 · p. 4, §3.3.2 · p. 4
3.3.3. Effectiveness of data augmentation
- Combining EMDA and VTLP outperformed either method alone at every tested augmentation size.The comparison tested 10k, 20k, 30k, and 40k augmented samples.Source: Figure 4 · p. 4, §3.3.3 · p. 4
3.3.4. Effects of Multiple Instance Learning
- MIL did not improve performance overall, although 2-second MIL inputs performed as well as or slightly better than large-field single-instance learning.Two 2-second instances provided the same total input length as a 4-second single instance while using fewer parameters; the authors suggest this may explain the result.Source: Table 4 · p. 4, §3.3.4 · p. 4
4. Conclusions
The conclusion presents end-to-end CNNs for modeling multi-second acoustic events and data augmentation for training with limited data. Experiments support the proposed methods, and future work is directed toward video applications.
- The proposed CNN architectures model several-second signals directly to learn AED end-to-end.Source: §4 · p. 4
- The proposed data augmentation method prevents overfitting and improves performance when training data is limited.Source: §4 · p. 4
- The experiments validate deeper architectures, large input fields, and data augmentation; future work will apply AED to video segmentation and summarization.Source: §4 · p. 4