Source-linked AI summary
AENet: Learning Deep Audio Features for Video Analysis
Naoya Takahashi, Michael Gygli, Luc Van Gool
TL;DR
Audio-event recognition is difficult because non-speech sounds come from varied sources and often require extended temporal analysis, while conventional audio features can lose temporal structure or remain task-specific. The paper introduces AENet, a large-input CNN trained end-to-end with data augmentation, and transfers its learned features to video analysis. AENet improves audio-event recognition and substantially benefits action recognition and video highlight detection, including 56.6% mAP for highlight detection.
Problem
Non-speech audio events have varied sources and extended temporal structure, while conventional representations can discard temporal order or remain task-specific.
Method
AENet uses large-input CNN architectures trained end-to-end with a diverse audio-event dataset and Equalized Mixture Data Augmentation, then extracts normalized activations as audio features.
Results
AENet features outperform baselines for action recognition and achieve 56.6% mAP for video highlight detection, significantly outperforming the baselines.
Takeaways & Limitations
The learned AENet representation generalizes from audio-event recognition to action recognition and video highlight detection without fine-tuning.
Abstract
from arXiv · showhide
We propose a new deep network for audio event recognition, called AENet. In contrast to speech, sounds coming from audio events may be produced by a wide variety of sources. Furthermore, distinguishing them often requires analyzing an extended time period due to the lack of clear sub-word units that are present in speech. In order to incorporate this long-time frequency structure of audio events, we introduce a convolutional neural network (CNN) operating on a large temporal input. In contrast to previous works this allows us to train an audio event detection system end-to-end. The combination of our network architecture and a novel data augmentation outperforms previous methods for audio event detection by 16%. Furthermore, we perform transfer learning and show that our model learnt generic audio features, similar to the way CNNs learn generic features on vision tasks. In video analysis, combining visual features and traditional audio features such as MFCC typically only leads to marginal improvements. Instead, combining visual features with our AENet features, which can be computed efficiently on a GPU, leads to significant performance improvements on action recognition and video highlight detection. In video highlight detection, our audio features improve the performance by more than 8% over visual features alone.
I. INTRODUCTION
The paper develops generic deep audio features for consumer-video analysis because conventional audio representations lose temporal information or are too task-specific. AENet combines large-input CNNs, data augmentation, and a diverse audio-event dataset, then evaluates transfer to video tasks.
- Motivation: Audio complements visual information in video analysis because it is not limited to the camera’s line of sight.The paper frames audio as an important component for understanding video scenes.
- Motivation: Conventional audio features are low-level or speech-oriented, while BoAW discards the temporal order of frame-level features.These limitations motivate representations tailored to non-speech audio events.
- Approach: AENet uses CNN architectures with up to 9 layers and a large input field to model entire audio events and train end-to-end.The paper targets audio events that may span several seconds.
- Contributions: The paper proposes data augmentation, a varied audio-event dataset, and publicly available pretrained models for generic feature extraction.The dataset contains sound events that may occur in consumer videos.
- Evaluation: The learned features are evaluated on audio event recognition, action recognition, and video highlight detection.The experiments assess both the source audio task and transfer to consumer-video tasks.
II. RELATED WORKS
Prior video-analysis methods learn visual representations from spatio-temporal interest points or CNNs that combine information across frames.
- Traditional video representations: Traditional video analysis used spatio-temporal interest points described with features such as SIFT, HOG, and HOF.
- CNN-based video analysis: CNN-based approaches investigated fusing spatial features over multiple video frames in time.
- CNN-based video analysis: Trajectory pooling was combined with CNN features in subsequent video-analysis methods.
B. Transfer learning
Transfer-learning work motivates reusable representations, but audio datasets and learned models have often been narrow or task-specific. The paper therefore targets a generic audio representation that can support multiple analysis tasks.
- Transfer-learning motivation: CNN representations trained on large datasets such as ImageNet can improve performance across many transfer-learning tasks.
- Traditional AER: Traditional AER methods model frame-level descriptors with GMMs or aggregate them using BoAW, but BoAW discards temporal order and hand-crafted pipelines are not end-to-end.
- Deep AER: DNN-based AER methods jointly learn representations and classifiers, but the cited approaches remain task specific.
- Research gap: The paper identifies a need for a generic audio representation that can support multiple audio-analysis tasks.
III. DEEP AUDIO FEATURE LEARNING
The paper uses CNNs with large temporal inputs to model audio events directly, preserving longer-range structure that conventional short-frame methods aggregate. It also introduces Equalized Mixture Data Augmentation to increase variation for these parameter-heavy networks.
- Deep convolutional architecture: Large-input CNNs model audio events over several seconds instead of first modeling only short speech-like frames.The architecture leverages CNN translation invariance to handle events that may occur at different positions in a long input.
- Deep convolutional architecture: The network is trained end-to-end to estimate audio-event class posteriors from large temporal inputs.The figure describes direct modeling of several seconds of audio and posterior-probability output.
- Deep convolutional architecture: The deeper CNN architecture uses multiple convolutional layers with 3×3 kernels unless otherwise specified.Table I identifies the deeper CNN architectures and their default kernel size.
- Data augmentation: Equalized Mixture Data Augmentation creates synthetic same-class samples by mixing delayed, equalized source signals.Random mixing, delay, and parametric-equalizer settings generate additional samples.
D. Dataset
The dataset is designed to support generic audio-feature learning by using diverse, lower-level sound-event classes rather than highly semantic mixtures. It combines harvested Freesound samples with additional classes and addresses noisy web data using multiple instance learning.
- Dataset design: The dataset targets discriminative, generic audio features by avoiding small class sets and high-level classes containing mixtures of different sounds.The design distinguishes individual events such as Church Bell from broad semantic situations such as Birthday Party.
- Dataset construction: Samples are harvested from Freesound and contain realistic variation from irrelevant tags, recording devices, noise, and brief event intervals.These properties make the database challenging while keeping its sounds similar to those in consumer videos.
- Multiple instance learning: Multiple instance learning replaces the final softmax with an aggregation layer over replicated CNN outputs for noisy bags of instances.The method supports max and Noisy OR aggregation while using labels available only at the bag level.
- Multiple instance learning: The MIL assumption is that a positive bag contains at least one positive instance, whereas a negative bag contains only negative instances.The probability that all instances in a bag are negative decreases exponentially as the number of instances increases.
IV. ARCHITECTURE VALIDATION AND AUDIO EVENT RECOGNITION
The audio event recognition experiments evaluate the proposed CNN architectures and augmentation method before using the learned representations for downstream video-analysis tasks.
- Architecture validation and audio event recognition: Audio event recognition is used to validate the proposed method and select an appropriate network architecture for producing AENet features.The evaluation assumes that more discriminative audio-event models provide more discriminative features for subsequent video tasks.
A. Implementation details
The experiments compare deeper CNN architectures and baselines for audio event recognition, using train/test splitting and varying input patch lengths. The implementation evaluates large-field CNNs against BoAW and CNN/DNN-HMM systems, with patch-length effects examined separately.
- The standard input used 400 frames, or 4 seconds, with 25 ms frames and 10 ms shifts; training randomly cropped 4-second samples.
- The data was randomly split into 75% training and 25% test sets, with irrelevant test sounds omitted after manual checking.
- The first experiment compared deeper CNN architectures with BoAW, HMM+DNN/CNN, and classical DNN/CNN baselines using a large input field.
- Input patch lengths of 50, 100, 200, 300, and 400 frames were tested, corresponding to 0.5–4 seconds.
- Figures 3 and 4 report performance changes across input patch lengths and data-augmentation methods, respectively.
D. Effectiveness of data augmentation
The experiments assess augmentation and multiple-instance learning as strategies for improving robustness with limited or noisy audio-event data. Combining augmentation methods was consistently stronger, while MIL did not improve the main configuration.
- Using both EMDA and VTLP always outperformed using either augmentation method alone across augmented-sample counts of 10k–40k.
- Combining EMDA and VTLP creates complementary data variation that helps train a more robust and general model from limited data.
- MIL did not improve performance for the evaluated noisy database and large-input configurations.
- MIL with a medium 2-second input field performed as well as the corresponding 4-second configuration while using fewer parameters.
- The MIL and normal-training results are summarized in Table IV.
B. Action recognition
AENet features were evaluated for action recognition by combining them with visual features and comparing them with MFCC, BoAW, and visual-only baselines. The proposed features improved recognition without fine-tuning and generally helped across classes.
- AENet features were compared with visual-only, MFCC, and BoAW baselines using C3D visual features and a linear multiclass SVM.
- The evaluation used 6,837 videos containing audio from 51 action categories after excluding videos without audio.
- AENet features significantly outperformed all baselines for action recognition without fine-tuning.
- MFCC audio features produced no considerable gain over visual features alone, whereas AENet features yielded more discriminative and general audio representations.
- Adding AENet features improved or preserved performance for most classes, reducing confusion between Playing Dhal and Playing Cello but slightly increasing confusion with Playing Daf.
F. Video Highlight Detection
The highlight-detection experiments combine C3D visual features with AENet audio features and learn scores using ranking-based objectives. The training setup addresses noisy highlight labels and compares positive and negative moments within each video.
- The highlight dataset contains six domains and about 100 YouTube videos per domain, totaling 1,430 minutes split into training and testing halves.
- Figure 5 compares confusion matrices for C3D and C3D+AENet, with positive diagonal values indicating class improvements and negative off-diagonal values indicating increased misclassification.
- Highlights were labeled from edited videos, while moments appearing only in the corresponding raw videos were labeled non-highlights.
- C3D visual features were concatenated with AENet features, and a higher neural-network H-factor indicated highlight moments.
- The ranking loss requires highlight moments to score above non-highlight moments within each video without constraining absolute scores.
- Huber loss reduces the effect of ambiguous margin violations, while multiple-instance ranking requires only the highest-scoring highlight segment to rank above the negative.
I. Baselines
The action-recognition evaluation compares AENet-related audio choices against visual-only and conventional audio baselines. It uses C3D features alone, C3D with MFCC, and BoAW as baselines.
- The evaluation compares C3D features alone, C3D with MFCC, and BoAW as baselines for action recognition.
- MFCC features are averaged within each moment before evaluation.
- BoAW features are calculated separately for each moment.
J. Results
AENet features outperform or match competing audio features across most highlight-detection domains, reaching 56.6% mAP overall. They improve detection in domains with characteristic highlight sounds, while learned audio features generalize to video analysis.
- 56.6% mAP overall, with AENet features significantly outperforming the baselines and the current state-of-the-art.The metric is mean average precision averaged over all test-set videos.
- AENet features improve highlight detection for skating, surfing, skiing, and parkour, where characteristic sounds occur at highlight moments.Examples include stunts, surfing starts, footsteps, and other impulsive sounds.
- MFCC and BoAW can miss or incompletely characterize impulsive sounds, whereas AENet features better capture these sounds in parkour.
- 8.6% average improvement over C3D visual features is reported for domain-specific highlight detection.
- The learned audio features generalize to video analysis and produce superior action-recognition and video-highlight-detection performance compared with commonly used audio features.