PaperlayerView exact revision on arXiv

Source-linked AI summary

Sound Event Detection: A Tutorial

Annamaria Mesaros, Toni Heittola, Tuomas Virtanen, Mark D. Plumbley

arXiv:2107.05463v1eess.AS

TL;DR

Sound event detection addresses the problem of identifying what sounds occur and when in an audio signal, despite application-specific classes and limited strongly labeled data. This tutorial synthesizes SED definitions, signal processing, machine-learning methods, evaluation, and future perspectives. It describes DNNs as the predominant approach, while weak labels, transfer learning, and varied representations address data constraints.

  • Problem

    Sound event detection must identify the temporal activity of sounds in audio, but applications use different target classes and lack universally applicable datasets or models.

  • Method

    The tutorial presents SED through its definitions, signal representations, machine-learning approaches, data strategies, evaluation options, and future perspectives.

  • Results

    Deep neural networks are predominant in SED, while weak labels, transfer learning, and advanced training strategies address shortages of strongly labeled data.

  • Takeaways & Limitations

    Engineered time-frequency representations remain common because end-to-end approaches require substantially larger training datasets, and advanced strategies help cope with limited strongly labeled data.

Abstract

from arXiv · show

The goal of automatic sound event detection (SED) methods is to recognize what is happening in an audio signal and when it is happening. In practice, the goal is to recognize at what temporal instances different sounds are active within an audio signal. This paper gives a tutorial presentation of sound event detection, including its definition, signal processing and machine learning approaches, evaluation, and future perspectives.

I. SOUND EVENTS IN OUR EVERYDAY ENVIRONMENT

Sound event detection seeks to identify which sounds are active and when within complex auditory scenes. The task is difficult because natural environments contain diverse, overlapping, application-specific sounds without a universal class ontology or model.

  • Human listeners recognize complex soundscapes by learning associations among sounds, their sources, and their names.
  • Sound event detection estimates the temporal instances at which different sounds are active in an audio signal.
  • Target classes vary by application, commonly including environmental sounds such as birdsong, passing cars, and footsteps.
  • Events differ in duration and acoustic structure, may be integrated from elementary components, and can be quieter than competing sounds at the microphone.
  • Polyphonic environments allow multiple sound events to overlap without predefined co-occurrence rules, requiring relationships to be learned from data statistics.
  • The unlimited number of possible classes and ambiguous category definitions prevent universally applicable datasets or models, so systems must meet application-specific needs.

III. THE GENERAL MACHINE LEARNING APPROACH FOR SOUND EVENT DETECTION

The general SED approach frames detection as multi-label classification over consecutive short audio segments, using learned features to estimate class activity over time. Deep neural networks became predominant because simultaneous outputs naturally represent overlapping sound classes and have improved state-of-the-art performance.

  • Supervised SED learns an acoustic model from audio recordings and binary temporal annotations indicating whether each target class is active.
  • Multi-label classification produces an activity estimate for every considered class in each short consecutive audio segment.
  • Segment length determines output resolution, which may range from analysis frames of typically 20–100 ms to longer intervals.
  • Temporal context from consecutive frames helps model contiguous event instances, while postprocessing can combine or reject implausibly short detections.
  • Early SED systems borrowed GMMs and HMMs from speech and music analysis, whereas modern DNNs handle simultaneous class activity through multiple output neurons.
  • DNNs have become predominant in SED and are associated with major gains in state-of-the-art performance and problem complexity.

IV. DATA

SED data must represent the target classes, acoustic variability, and temporal activity relevant to a specific application. Strong labels provide event timing but are costly, while weak, noisy, or synthetic data offer scalable alternatives with important quality and realism trade-offs.

  • SED models require training audio from conditions similar to intended use, with annotations covering the target sound events.
  • Data collection must cover variability in sources, source states, environments, and interfering sounds, although complete coverage is impossible.
  • Sufficient training-data diversity can lead to good generalization of trained models.
  • Strong labels specify onset and offset times for individual events, whereas weak labels indicate only that a sound occurs somewhere in a longer recording.
  • Target classes should be selected in advance because annotating all audible sounds can produce diverse labels with too few examples for some classes.
  • Crowdsourced weak annotations scale data collection but can be noisy because non-experts may choose incorrect or missing labels.
  • Synthetic mixtures enable fast production of accurate strong labels, but may create unnatural event combinations or miss real-environment complexity.
  • Manually annotated datasets are smaller than weakly labeled or crowdsourced datasets, with AudioSet representing the large-data extreme described here.

V. SIGNAL PROCESSING METHODS FOR SOUND EVENT DETECTION

SED systems combine feature representation and classification, with preprocessing and data augmentation often supporting training. Augmentation expands acoustic variability and creates new overlapping-event combinations while preserving annotation correctness.

  • SED systems use feature representation and classification, with training and testing phases for learning acoustic models and predicting event activity.Preprocessing and data augmentation may also be included.
  • Data augmentation artificially increases training-data amount and diversity when recordings are difficult to obtain.
  • Augmentation can create new combinations of overlapping events that may be absent from naturally collected polyphonic recordings.
  • Common SED augmentations include time stretching, pitch shifting, dynamic-range compression, impulse-response convolution, block mixing, mixup, time shifting, and noise addition with varied SNR.
  • Augmented annotations must be transformed with the audio so event labels remain correct; time stretching extends both audio duration and class activations.

B. Feature representation

SED feature representations commonly use log mel energies, which encode signal energy on perceptually motivated scales. Feature design balances frequency resolution, temporal resolution, computational cost, and the needs of diverse sound events.

  • Log mel energies are commonly used features that represent audio energy on perceptually motivated frequency and amplitude scales.
  • Mel spectra redistribute spectrogram energy through a filterbank on a perceptual pitch scale relating frequency to perceived pitch.
  • Triangular mel filters compute each band’s energy as a weighted sum of spectral amplitudes within the filter range.
  • More mel filters can help resolve overlapping sounds at closely spaced frequencies, enabled by models that learn from higher-dimensional features.
  • DNN-based SED no longer requires DCT decorrelation because neural networks can learn from correlated features, while model size trades accuracy against computational complexity.
  • A basic SED system can use 40 log mel energies computed in 40 ms windows as a compromise between detailed representation, dimensionality, and varied event durations.
  • Alternative representations include harmonic-percussive decompositions and constant-Q transforms, with separate models potentially helping when different sound types overlap.
  • Feature learning can learn time-frequency representations from data, but raw-audio feature learning requires large datasets and has not matched reduced-dimensionality representations in the cited examples.

VI. MACHINE LEARNING FOR SOUND EVENT DETECTION

Deep neural networks are now the dominant approach in environmental sound analysis and classification, but SED’s shortage of strongly labeled data remains a central challenge. Weak labels and transfer learning provide strategies for addressing this data limitation.

  • Deep neural networks have become the dominant approach in environmental sound analysis and classification after producing major improvements across domains.
  • DNNs require large training datasets, while SED lacks large strongly labeled datasets.
  • Training strategies using weak labels and transfer learning can address shortcomings in strongly labeled SED data without dramatically changing general system architectures.

A. Convolutional Recurrent Neural Networks

CRNNs combine convolutional feature extraction with recurrent temporal modeling to estimate sound-event activity probabilities over time. Their outputs are trained, thresholded, and postprocessed into event activity sequences or instances.

  • A. Convolutional Recurrent Neural Networks: A CRNN receives a time-frequency representation and uses convolutional layers to learn discriminative features from it.Convolutional blocks combine convolution, nonlinear activation, and pooling.
  • A. Convolutional Recurrent Neural Networks: Recurrent layers model temporal dependencies, including how events start, end, follow one another, and evolve internally.Bidirectional recurrent units use both past and future inputs.
  • A. Convolutional Recurrent Neural Networks: In the illustrated CRNN, pooled convolutional features are stacked across frequency before entering recurrent layers.The example maps 2x128xT features to a 256xT matrix.
  • A. Convolutional Recurrent Neural Networks: Feed-forward output layers produce individual probabilities for each target event class using sigmoid units.
  • A. Convolutional Recurrent Neural Networks: The output sequence is binarized using a threshold such as 0.5 or an optimized problem-specific threshold, then postprocessed into event instances.Heuristics can enforce plausible event lengths and activation patterns.
  • A. Convolutional Recurrent Neural Networks: SED networks are trained by backpropagation and gradient-based optimization using a cost function measuring prediction error against target outputs.Cross-entropy is used in SED training.
  • A. Convolutional Recurrent Neural Networks: Dropout regularizes large networks by randomly ignoring units during training, making models more robust to small changes.
  • A. Convolutional Recurrent Neural Networks: Architecture and hyperparameters are adjusted for task requirements and performance, with recurrent layers used when temporal sequences must be modeled.Randomized search can reduce the cost of hyperparameter selection compared with exhaustive grid search.

B. Advanced methods

Advanced SED methods address limited annotated data through transfer learning, weak or noisy labels, and student-teacher knowledge distillation. These approaches reuse external representations, learn from coarse labels, or transfer behavior between networks.

  • Transfer learning: Transfer learning uses data from a pretext task to learn embeddings that serve as input features for a downstream SED task.Examples of pretrained networks include VGG-ish, SoundNet, and L3-Net.
  • Transfer learning: L3-Net learns acoustic embeddings from audio-video correspondence and later uses its trained audio branch to produce features.The network has separate audio and video branches whose convolutional outputs are flattened, concatenated, and passed to fully connected layers.
  • Weak and noisy labels: Weak labels omit temporal information, whereas noisy labels may be incorrectly attributed; both create challenges for training SED systems.Multiple instance learning treats frames as instances grouped into weakly labeled bags, with instance predictions pooled to bag-level outputs.
  • Student-teacher methods: Student-teacher methods transfer knowledge from a teacher network to a student, including solutions that use coarse tagging to guide finer boundary detection.The teacher can provide outputs instead of reference targets, while unlabeled data supports learning.

VII. PERFORMANCE EVALUATION

SED performance is evaluated by comparing system outputs with reference annotations using segment-based or event-based views and classification metrics. These choices reflect different temporal representations and application requirements, while current metrics retain important limitations.

  • Evaluation principles: Evaluation metrics compare system output with reference annotations to count correct and erroneous detections.The paper notes that user satisfaction would ideally measure performance, but development commonly uses computational metrics.
  • Segment-based evaluation: Segment-based evaluation quantizes outputs and annotations on a temporal grid, then compares binary activity indicators for each class and segment.The process counts true positives, true negatives, false positives, and false negatives.
  • Classification metrics: Precision, recall, F-score, and error rate are common classification metrics derived from correct and erroneous detection counts.Error rate additionally represents paired false-positive and false-negative occurrences as substitutions, with remaining errors counted as insertions or deletions.
  • Event-based evaluation: Event-based evaluation compares detected and annotated sound instances by checking label agreement and onset or offset alignment within a permitted collar.Figure 8 illustrates a 200 ms collar applied to both onset and offset.
  • Evaluation views: Segment-based evaluation measures active temporal regions, whereas event-based evaluation measures individual sound-event instances.Because practical methods often operate at segment level while SED is defined at event level, metric choice depends on application requirements.
  • Metric limitations: The field still needs more suitable metrics because collars weaken event-based evaluation and segment-based evaluation only partly alleviates boundary subjectivity.PSDS is one proposal that defines true and false positives using system-reference intersection and tolerates event-instance segmentation.

VIII. RELATED RESEARCH PROBLEMS

SED shares features and machine-learning techniques with speech and music analysis but differs in temporal labeling, overlapping sources, and sound co-occurrence. These differences motivate adaptations such as multi-label outputs, segment mixing, and localization extensions.

  • Relation to classification: Unlike whole-recording classification, SED estimates temporal activity for sound classes and therefore requires outputs at consecutive time points.Classifier architectures can be adapted by omitting temporal pooling operations.
  • Polyphonic detection: SED commonly targets multiple overlapping sources, requiring polyphonic output rather than the single-source focus typical of automatic speech recognition.Multi-label classifiers with sigmoid output nonlinearities can represent simultaneous sound classes.
  • Data augmentation: Audio augmentation methods from other domains apply to SED, while mixing existing segments and annotations is especially useful because many sound combinations overlap.The paper identifies overlapping combinations as a distinctive feature of SED augmentation.
  • Acoustic features: Log mel energies used in speech analysis can be directly applied to SED because target events include transient, harmonic, and noise-like components.These features are motivated by perceptual frequency and amplitude scales.
  • Localization: Multichannel SED can extend detection toward localization, including moving targets and association of estimated sound classes with directions.The paper focuses on single-channel methods but identifies multiple microphones as a potential route to improved accuracy and joint localization.

IX. FUTURE PERSPECTIVES

Future SED research addresses data scarcity, privacy, transfer to unseen classes, robustness to deployment mismatch, and improved benchmarking. The paper presents human-in-the-loop, federated, zero-shot, adaptation, and evaluation strategies for these challenges.

  • Data and annotation: Data scarcity is a bottleneck because deep learning methods require large annotated datasets, motivating alternatives to conventional supervised learning.The paper frames alternative acoustic-model construction as a response to limited suitable annotations.
  • Data and annotation: Active learning selects unlabeled audio for annotation where labeling is expected to improve model accuracy most.One reported system required only 2% of the annotations used by standard supervised learning to reach the same accuracy.
  • Privacy: Federated learning seeks to improve a general model using information from multiple users without transferring or storing their audio in the cloud.This addresses privacy concerns arising when end-user audio contains personal information such as conversation.
  • Transfer to new classes: Zero-shot learning uses textual descriptions or other auxiliary information about unseen sound classes to obtain acoustic models without audio samples from those classes.The paper describes extending zero-shot temporal classification to SED.
  • Robustness: SED systems are sensitive to training-test mismatches, including deployment on different devices, so model adaptation is used to retain performance in new conditions.The paper identifies robustness to changing conditions as an ongoing research concern.
  • Benchmarking: DCASE provides recurring public tasks, open data, benchmarking, and technical descriptions to support reproducible comparison of SED methods.The paper argues that new benchmarking methods should address emerging limitations as the field progresses.

APPENDIX: NEURAL NETWORKS FOR AUDIO

Neural networks learn mappings from audio-derived inputs to outputs through layered nonlinear units and gradient-based training. CNNs extract local time-frequency features, RNNs model temporal context, and CRNNs combine both for sound event detection.

  • Neural-network foundations: Feed-forward neural networks map an input vector x to an output ŷ, with parameters θ estimated from example training pairs.Layers of nonlinear units allow the model to represent complex relationships between inputs and outputs.
  • Neural-network foundations: Gradient-based learning minimizes a cost function, while backpropagation computes gradients by propagating information backward through the network.The layered nonlinear structure makes direct gradient computation difficult.
  • Output representations: For multi-class multi-label tasks, separate sigmoid outputs estimate independent class probabilities, supporting audio tagging and sound event detection.Unlike softmax outputs for single-label classification, these probabilities do not need to sum to one.
  • Convolutional networks: CNNs apply small kernels to two-dimensional time-frequency representations, using convolutions, nonlinear activations, and pooling to learn feature representations.A typical kernel may be 3 by 3, and the convolution produces an output matrix from the input matrix and kernel.
  • Convolutional networks: SED CNNs commonly pool only across frequency and preserve time resolution, whereas audio classification can subsample both frequency and time.Preserving the time axis provides decisions for consecutive audio segments needed to estimate temporal activities.
  • Recurrent and hybrid networks: RNNs use the previous hidden state to process sequential audio, enabling online predictions from past inputs and bidirectional offline processing with future context.CRNNs combine convolutional feature extraction with recurrent temporal modeling for sound event detection.
Loading 2107.05463v1…