PaperlayerView exact revision on arXiv

Source-linked AI summary

Audio Event Detection using Weakly Labeled Data

Anurag Kumar, Bhiksha Raj

arXiv:1605.02401v3cs.SDcs.AIcs.MM

TL;DR

Audio event detectors typically require scarce, costly strong annotations, while much multimedia data has only weak labels without event timing. The paper formulates detection as multiple-instance learning and proposes SVM- and neural-network-based frameworks. The methods detect and temporally localize events, achieving mean AUC around 0.7 with limited training data.

  • Problem

    Strongly labeled audio data specify event segments, but multimedia recordings generally provide weak labels without event counts, locations, or durations.

  • Method

    The paper formulates weakly labeled audio event detection as multiple-instance learning and proposes support-vector-machine and neural-network frameworks.

  • Results

    Mean AUC around 0.7 demonstrates reasonable performance for detection and temporal localization with limited training data.

  • Takeaways & Limitations

    The framework reduces dependence on strongly labeled data while providing temporal event locations absent from the original weak labels.

  • Takeaways & Limitations

    The framework still requires evaluation on a larger set of events and investigation of more effective multiple-instance learning methods.

Abstract

from arXiv · show

Acoustic event detection is essential for content analysis and description of multimedia recordings. The majority of current literature on the topic learns the detectors through fully-supervised techniques employing strongly labeled data. However, the labels available for majority of multimedia data are generally weak and do not provide sufficient detail for such methods to be employed. In this paper we propose a framework for learning acoustic event detectors using only weakly labeled data. We first show that audio event detection using weak labels can be formulated as an Multiple Instance Learning problem. We then suggest two frameworks for solving multiple-instance learning, one based on support vector machines, and the other on neural networks. The proposed methods can help in removing the time consuming and expensive process of manually annotating data to facilitate fully supervised learning. Moreover, it can not only detect events in a recording but can also provide temporal locations of events in the recording. This helps in obtaining a complete description of the recording and is notable since temporal information was never known in the first place in weakly labeled data.

1. INTRODUCTION

The paper addresses the challenge of analyzing rapidly growing multimedia collections whose metadata is sparse and whose strong audio annotations are scarce. It proposes learning acoustic event detectors from weak labels through multiple-instance learning, while also estimating event timing.

  • Motivation: Rapidly growing consumer multimedia collections are largely unannotated or described only by cryptic, high-level metadata.This creates a need to derive content descriptions automatically.
  • Applications: Automatic sound detection supports content organization and can capture information carried by audio that video may not provide.The paper also notes applications including surveillance, wildlife monitoring, context recognition, and health monitoring.
  • Annotation challenge: Strongly labeled audio data specify event-containing segments and negative examples, but such annotations are generally scarce, time-consuming, and expensive to create.Weak labels typically indicate presence or absence without event counts, times, or durations.
  • Proposed direction: The paper treats recordings as collections whose event-bearing regions share consistency, enabling detectors to be learned from weakly labeled data.The approach keys on consistency among regions in recordings tagged with an event.
  • Contribution: The proposed multiple-instance learning framework learns acoustic event detectors from weak labels and assigns temporal locations to events in test recordings.Temporal localization is obtained even though the original weak labels did not provide event timing.

2. RELATED WORK

Prior audio event detection generally uses supervised learning with clearly demarcated positive and negative segments, while weakly labeled multimedia remains underexplored. This paper formulates weakly labeled audio event detection as a multiple-instance learning problem.

  • Audio event detection: Audio-based multimedia analysis detects acoustic events such as clapping, cheering, and gunshots, with surveillance among its prominent applications.Other applications include automatic indexing of multimedia recordings of games.
  • Prior methods: Generic real-world sound-event detection has been restricted by the lack of large-vocabulary open-source datasets.Earlier work modeled sound events with Gaussian-mixture models in architectures similar to speech-recognition GMM-HMM systems.
  • Supervised learning: Conventional detectors are trained with clearly demarcated positive and negative audio instances, requiring human annotation that is time-consuming and expensive.These methods therefore rely on strongly labeled examples of event-containing and event-absent segments.
  • Weak annotations: Weak annotations indicate event presence in recordings without specifying the number or locations of instances, and are generally easier to obtain or generate.This makes them a potential alternative source of training information.
  • Research gap: Research on weakly labeled event detection has focused mainly on visual content, while generic audio event detection from weak labels has received almost negligible attention.The paper positions its contribution within this comparatively sparse audio literature.
  • This paper: The paper formulates audio event detection using weak labels as multiple-instance learning, where learning uses labels for ensembles of instances rather than individual instances.The framework can use established MIL adaptations such as support vector machines and k-nearest neighbors.

3. AED USING WEAKLY LABELED DATA

The formulation treats each audio recording as a weakly labeled collection and its segments as instances, with labels indicating which events may occur somewhere in the recording. Conventional supervised learning cannot directly isolate event segments when their timing is unknown.

  • Problem formulation: The objective is to train event models from recordings labeled only for event presence or absence, without event locations or occurrence counts.The target events include examples such as clapping, barking, and cheering.
  • MIL representation: Each recording is represented as a bag, and its audio segments are represented as instances within that bag.The paper defines a collection of recordings and a set of events for which detection models are built.
  • Weak labels: A recording’s weak label specifies a subset of events known to be present, and the subset may be empty.For example, one recording may be labeled as containing E1, E3, and E6.
  • Challenge: Using all recordings tagged with an event is insufficient because those recordings may also contain other events and do not reveal the target event’s start or end times.Consequently, the specific event-containing segment cannot simply be extracted for supervised training.

4. PROPOSED FRAMEWORK

The proposed framework casts weakly labeled audio event detection as a multiple-instance learning problem. In this formulation, labels apply to collections of instances rather than to individual instances, allowing suitable MIL algorithms to be used.

  • Framework formulation: Weakly labeled audio event detection is formulated as a multiple-instance learning problem.The paper presents MIL as a generalized form of supervised learning with labels available for collections of instances.

4.1 Multiple Instance Learning

Multiple Instance Learning (MIL) learns instance-level classifiers from labels attached to bags rather than individual instances. Positive bags contain at least one positive instance, while negative bags contain only negative instances.

  • MIL attaches labels to bags of instances instead of labeling each individual instance.The training data are bag-label pairs, although the learned classifier classifies individual instances.
  • A positive bag contains at least one target-class instance, whereas a negative bag contains negative instances only.
  • The learning task is to train a classifier that predicts the label of a new bag from training bag-label pairs.
  • MIL models the relationship between bag labels and unobserved instance labels through constraints.For a positive bag, at least one instance must be positive; instances in negative bags are negative.
  • The paper uses mi-SVM and neural networks as two frameworks for solving the MIL problem.

4.2 MIL for SVM (mi-SVM)

The mi-SVM framework adapts SVM learning to weakly labeled bags by treating positive-bag instance labels as unknown variables and iteratively refining them with the classifier.

  • mi-SVM operates at the instance level and maximizes the margin of individual instances from a linear discriminant.The paper uses mi-SVM rather than the alternative MI-SVM, which maximizes the margin of bags.
  • Positive-bag instance labels are unknown integer variables, while all instances in negative bags are constrained to be negative.
  • The mi-SVM optimization jointly seeks an instance labeling and a separating hyperplane that maximizes a soft margin.
  • The algorithm initializes instance labels, computes an SVM solution, updates positive-bag labels using discriminant scores, and repeats.If every instance in a positive bag becomes negative, the instance with the maximum discriminant value is assigned a positive label.
  • The difficult mixed integer problem is handled by alternating standard SVM optimization with imputation of labels for positive-bag instances.The two steps repeat until the imputed labels no longer change.

4.3 MIL of Neural Networks (BP-MIL)

BP-MIL adapts neural-network training to bag-level supervision by defining each bag’s output from its highest-scoring instance and updating weights accordingly.

  • BP-MIL modifies neural-network training so divergence is computed from bag-level labels rather than instance-specific labels.
  • The network represents a bag using the instance with the maximum output, with the bag target set to its observed label.Positive bags have target 1, whereas negative bags have target 0.
  • For negative bags, minimizing the bag divergence pushes the outputs of all instances toward 0.The ideal case has zero output for every input in the negative bag.
  • For positive bags, minimizing divergence pushes at least one instance output toward 1.The ideal case has one or more inputs in the bag producing output 1.
  • Training processes complete bags before updating network parameters using gradients determined by the maximal-output instance.After training, predictions can be made instancewise, while a bag label can be based on its maximal-scoring instance.

4.4 MIL for AED using weakly labeled data

Weakly labeled recordings are converted into MIL bags by segmenting them into short audio instances. Recording-level event labels then support detector training and event localization within recordings.

  • Each recording is segmented into short, potentially overlapping audio segments that become instances in its MIL bag.The number of segments depends on recording duration, segment length, and overlap.
  • A recording labeled as containing an event becomes a positive bag because at least one of its segments contains that event.
  • A recording labeled as not containing an event becomes a negative bag because none of its segments is positive for that event.
  • The resulting MIL representation enables models to predict event presence or absence for recordings and identify event locations within them.

4.5 Temporal Localization of Events

The MIL frameworks classify individual audio segments after learning from bag-level labels, enabling event detection at specific time segments and temporal localization in test recordings.

  • Instance-level classification: MIL frameworks learn from bag-level labels but classify individual recording segments after training.In audio analysis, recordings are bags and their segments are instances.
  • Temporal mapping: An event detected in a segment can be localized to that segment’s corresponding time interval in the recording.Segment duration is l seconds, and consecutive windows may be shifted by l′; non-overlapping segments use l′ = l.
  • Temporal localization: The framework provides temporal event locations even though temporal information was absent from the training labels.This supports a more complete description of recordings in terms of detected audio events.

4.6 GMM based features for audio segments

The paper represents short audio segments with GMM-derived features that combine coarse distributional information and finer mean structure before applying MIL event-detector models.

  • Feature motivation: MFCC vectors are converted into secondary representations because direct MFCC sequence characterization is ineffective for audio classification.The paper uses GMM-derived features for short audio segments.
  • Representation limitation: Bag-of-words features lose distributional detail in short segments, while enlarging the codebook can produce sparse, variable histograms.This motivates the GMM-based representation for fine-level event detection from weak labels.
  • F Features: The F feature is a G-dimensional normalized soft-count histogram describing how MFCC vectors distribute across background-GMM components.It replaces hard quantization with soft assignment and provides a coarser but robust representation.
  • M Features: The M feature adapts the universal GMM means to each segment using maximum a posteriori estimation, then concatenates the updated means.The resulting vector has G×D dimensions and gives a more detailed characterization of the segment distribution.
  • Combined representation: F is more robustly estimated, whereas M is more detailed but more sensitive to inter-instance variability.Together they represent coarse and fine signal structure; experiments use F alone or concatenate F and M.
  • MIL input: Segment-level feature vectors are organized into recording bags and passed to the BP-MIL or mi-SVM frameworks for detector learning.The input uses F alone or concatenated F and M vectors.

5. EXPERIMENTS AND RESULTS

The experiments evaluate multiple-instance learning for acoustic event detection on weakly labeled recordings, comparing configurations, supervised learning, and temporal localization. Results show effective recording-level detection and useful temporal localization despite no temporal information during training.

  • Experimental setup: The dataset contains 457 recordings spanning over 22 hours, with recordings averaging about 2.9 minutes.
  • Experimental setup: Each recording is treated as a bag, and one-second segments with 50% overlap form over 150,000 instances.
  • Evaluation: AUC evaluates detection performance at recording level and temporal localization, with higher values indicating better detection.
  • Recording-level detection: Weak-label miSVM performs worse than supervised SVM overall, but remains comparable for several events using only weak labels.
  • Recording-level detection: Increasing GMM components improves AUC by about 2−4% for several events, but has no benefit for some and decreases Children’s Voices performance by about 4%.
  • Recording-level detection: Adding M features improves miSVM for almost all events, with absolute gains of 10.3%, 11.4%, and 19.4% for Hammering, Laughing, and Marching Band.
  • Results: The best miSVM configuration achieves a mean AUC of 0.704 across events, while temporal localization shows about a 5% mean-AUC drop for both frameworks.

6. DISCUSSIONS AND CONCLUSION

The framework detects and temporally localizes acoustic events from weakly labeled data, achieving reasonable performance while reducing dependence on strongly labeled examples. The authors identify feature choice, event coverage, and more effective MIL methods as areas for improvement.

  • The framework learns acoustic event detectors from weakly labeled data and provides temporal event locations.
  • Clanking, Scraping, and Children’s Voices are easier to detect, whereas Drums, Hammering, and Laughing are harder for both methods.
  • Mean AUC of around 0.7 with limited training data demonstrates the proposed approach’s success across SVM and neural-network methods.Overall performance of both methods is similar.
  • Gaussian-mixture feature representations are effective for short audio segments but are not claimed to be optimal for audio event classification.Other feature representations may improve overall MIL performance.
  • Weakly labeled web multimedia can support event detection and temporal localization, with predicted instance labels also usable in active learning.
  • A larger event set and more effective multiple-instance learning methods remain needed to assess the framework more comprehensively and improve scalability.
Loading 1605.02401v3…