PaperlayerView exact revision on arXiv

Source-linked AI summary

Adaptive pooling operators for weakly labeled sound event detection

Brian McFee, Justin Salamon, Juan Pablo Bello

arXiv:1804.10070v2cs.SDcs.LGeess.ASstat.ML

TL;DR

Strong annotations make dynamic sound event detection costly, motivating a multiple instance learning formulation with static excerpt labels and pooled predictions. The paper introduces auto-pool, adaptive operators that interpolate among standard pooling strategies; across three datasets, they outperform non-adaptive pooling for static prediction and nearly match strongly supervised dynamic accuracy.

  • Problem

    Strong annotations specify source activity at every instant but are labor- and cost-intensive, limiting scalable sound event detection.

  • Method

    Auto-pool adaptively interpolates among pooling strategies while aggregating dynamic predictions for weakly labeled multiple instance learning.

  • Results

    Across three datasets, auto-pool methods outperform standard non-adaptive pooling for static prediction and nearly match models trained with strong annotations for dynamic prediction.

  • Takeaways & Limitations

    Weakly labeled multiple instance learning may provide comparable dynamic accuracy while reducing the effort and cost of acquiring training labels.

Abstract

from arXiv · show

Sound event detection (SED) methods are tasked with labeling segments of audio recordings by the presence of active sound sources. SED is typically posed as a supervised machine learning problem, requiring strong annotations for the presence or absence of each sound source at every time instant within the recording. However, strong annotations of this type are both labor- and cost-intensive for human annotators to produce, which limits the practical scalability of SED methods. In this work, we treat SED as a multiple instance learning (MIL) problem, where training labels are static over a short excerpt, indicating the presence or absence of sound sources but not their temporal locality. The models, however, must still produce temporally dynamic predictions, which must be aggregated (pooled) when comparing against static labels during training. To facilitate this aggregation, we develop a family of adaptive pooling operators---referred to as auto-pool---which smoothly interpolate between common pooling operators, such as min-, max-, or average-pooling, and automatically adapt to the characteristics of the sound sources in question. We evaluate the proposed pooling operators on three datasets, and demonstrate that in each case, the proposed methods outperform non-adaptive pooling operators for static prediction, and nearly match the performance of models trained with strong, dynamic annotations. The proposed method is evaluated in conjunction with convolutional neural networks, but can be readily applied to any differentiable model for time-series label prediction.

I. INTRODUCTION

Sound event detection requires dynamic labels, but strong annotations are costly, motivating weak-label learning through multiple instance learning and adaptive pooling.

  • Strongly labeled data specifies each source’s presence or absence at every instant, but its acquisition cost impedes scalable SED development.
  • Weakly labeled excerpts provide coarse labels, so dynamic predictions must be pooled into static predictions during training.
  • Auto-pool adaptively generalizes max-, mean-, and min-pooling for jointly training dynamic predictors from weak annotations.
  • The approach is evaluated across datasets with weak labels, including AudioSet and DCASE 2017, where weak-label SED became a prominent research problem.

C. Sound event detection using weakly labeled data

Weakly labeled SED combines dynamic instance prediction with temporal aggregation, but existing pooling and attention approaches have limitations for MIL instance-level outputs.

  • Weakly labeled SED methods pair an instance-level model with an aggregation method that converts frame or chunk predictions into bag-level outputs.
  • Max-based aggregation is common in MIL, while soft-max and mean pooling offer alternatives with different assumptions and optimization behavior.
  • Standard attention summarizes structured inputs, but its intermediate representations generally do not directly constitute instance predictions required by MIL.
  • The proposed methods use a constrained, interpretable attention mechanism tied to instance predictions, with one class-specific parameter interpolating between standard pooling operators.

III. METHODS

The method formulates weakly labeled learning as MIL: instance predictions are aggregated into bag predictions, with adaptive pooling intended to improve gradient-based training.

  • The method develops adaptive pooling operators to reduce dynamic label predictions to static predictions during training.
  • MIL assigns one label to a bag of instances and seeks to learn an instance classifier from those weak labels.
  • The paper focuses on deep neural networks whose instance outputs are thresholded likelihoods.
  • Max pooling makes a bag positive when any instance prediction is positive, so the bag prediction depends only on the largest instance-level prediction.
  • During training, model parameters are optimized by maximizing the likelihood of labeled bags, for example through binary crossentropy.

B. Max-pooling

Max-pooling passes gradient information only through maximizing instances, making MIL training sensitive to random early predictions. Soft-max pooling preserves max-like behavior while distributing responsibility across instances, but its weights become diffuse as bags grow.

  • Max-pooling: Max-pooling gives non-maximizing instances zero gradient contribution during parameter updates.Its sub-gradient can be viewed as a weighted average whose non-maximizing inputs receive weight 0.
  • Max-pooling: Random early instance predictions can make max-pooling sensitive to initialization, unstable, and difficult to deploy.Early parameter updates depend entirely on single, randomly selected instances.
  • Soft-max pooling: Soft-max pooling is continuously differentiable and assigns each instance responsibility proportional to its label likelihood.Positive predictions receive more influence, while negative predictions receive less.
  • Soft-max pooling: Soft-max pooling has limited capacity to concentrate on a small subset because each instance weight is Θ(1/m).As bag size grows, its weight bounds converge to 1/m and it behaves similarly to unweighted averaging.

D. Auto-pooling

Auto-pooling introduces a learnable scalar that adapts the aggregation behavior to the task. The operator continuously interpolates among mean-, soft-max-, max-, and approximate min-pooling behaviors.

  • Auto-pooling: A learnable scalar α lets auto-pooling adapt and interpolate between different pooling behaviors.The parameter is learned alongside the model parameters θ.
  • Auto-pooling: When α = 0, auto-pooling becomes an unweighted mean; when α = 1, it becomes soft-max pooling; and when α →∞, it approaches max-pooling.These settings provide a continuous path between standard aggregation operators.
  • Auto-pooling: For α ≤0, auto-pooling approximates min-pooling by assigning larger weights to smaller input values.As α →−∞, the largest weight is achieved by the smallest instance prediction.
  • Auto-pooling: Supporting min-like behavior relaxes the MIL assumption that a positive bag label must equal the maximum instance label.Conjunction-like behavior can represent bags that are positive when all instances are positive.

E. Constrained auto-pooling

Constrained auto-pooling controls how much weight any single instance may receive by bounding the adaptive parameter. A φ+ = 0.5 constraint yields the constrained auto-pool variant used thereafter.

  • Constrained auto-pooling: Auto-pooling can constrain α according to the maximum or minimum aggregation weight permitted for one instance.The bounds relate α to φ+, the maximum weight, and φ−, the minimum weight.
  • Constrained auto-pooling: When φ−= φ+ = 1/m, α = 0 and the operator recovers mean-pooling.The weight constraints are therefore tight at the uniform-weight case.
  • Constrained auto-pooling: φ+ = 0.5 gives α ≤ln(m −1), limiting the maximum weight assigned to one instance.This bound is selected because 0.5 is the minimal upper bound that permits max-pooling behavior under the stated decision rule.
  • Constrained auto-pooling: The paper calls auto-pooling with the φ+ = 0.5 bound imposed constrained auto-pool (CAP).CAP is the constrained variant used for the remainder of the article.

F. Regularized auto-pooling

Regularized auto-pooling discourages extreme adaptive parameters through a quadratic penalty rather than an explicit weight bound. This promotes mean-like behavior while retaining flexibility to learn max-pooling when needed.

  • Regularized auto-pooling: Regularized auto-pool applies a penalty to α to prevent excessive concentration on individual instances without explicitly bounding weights.The approach uses a quadratic penalty whose magnitude grows with α.
  • Regularized auto-pooling: The regularization promotes mean-like behavior while allowing the model to learn max-pooling behavior when necessary.Its soft constraint differs from constrained auto-pool’s explicit maximum or minimum weight bound.
  • Regularized auto-pooling: For multi-label formulations, the penalty generalizes to the squared Euclidean norm λ∥α∥2.Here, f(θ) denotes the learning objective and λ > 0 is the penalty coefficient.

G. Multi-label learning

The study evaluates adaptive pooling for multi-label sound event detection across three datasets with different annotation and event-duration characteristics. It reports static and dynamic precision, recall, and F1, while highlighting dataset-specific limitations and label distributions.

  • Multi-label learning: Auto-pooling applies a separate learned pooling operator to each class in multi-label prediction, allowing pooling strategies to adapt independently.The class-specific parameter vector is jointly learned with the model.
  • Experimental comparison: The experiments compare max, mean, soft-max, auto-pool, constrained auto-pool, and regularized auto-pool across URBAN-SED, DCASE 2017, and MedleyDB.Regularized auto-pool is evaluated with λ ∈ {10−2, 10−3, 10−4}.
  • Evaluation: Static and dynamic performance is measured with precision, recall, and F1, using multi-label metrics for bags and one-second segment-based metrics for dynamic predictions.F1-macro is the unweighted average of class-wise F1 scores.
  • URBAN-SED: URBAN-SED contains 10,000 ten-second soundscapes with nearly 50,000 events from 10 classes, and its computationally generated annotations are correct and complete.The dataset is an order of magnitude larger than the largest manually compiled strongly labeled SED dataset, but its events can be unnaturally truncated.
  • DCASE 2017: DCASE 2017 uses over 50,000 ten-second excerpts for weakly supervised smart-car SED, but its annotations may omit events or contain semantically overlapping classes.Its event-duration distribution is more natural than URBAN-SED’s and may influence auto-pool behavior.

3) MedleyDB:

MedleyDB provides time-varying instrument labels and supports stem-based mixture augmentation, while its long instrument activations make labels often constant over ten-second training examples. The study uses a fixed dynamic predictor followed by alternative temporal pooling functions to compare these effects.

  • MedleyDB: MedleyDB contains 122 multitrack recordings spanning varied musical genres and styles.
  • MedleyDB: Its isolated instrument stems enable alternate mixtures in which one of n instruments is removed and the remaining n−1 stems are remixed.The mixtures use coefficients supplied by the MedleyDB Python package.
  • Experimental setup: The experiment retains eight common sources and repeats evaluation over 10 artist-conditional random 80–20 train-test splits.MedleyDB has no predefined evaluation split.
  • Label durations: Most instruments remain active longer than the ten-second observation window, so training-example labels are often entirely on or entirely off.Figure 5 shows the instrument activation-duration distribution used to motivate this expectation.
  • Model architecture: The architecture combines a fine-resolution dynamic predictor with a temporal pooling layer that produces one static prediction per class.The predictor uses convolutional layers and outputs multi-label frame predictions before pooling.
  • Model architecture: The complete architecture is length-agnostic, although some pooling functions are affected by input length.The dynamic predictor and pooling layer are each agnostic to the number of input frames.

C. Training and evaluation

The experiments compare adaptive and non-adaptive pooling for static and dynamic SED prediction across three datasets. On URBAN-SED, adaptive pooling performs strongly, with CAP nearly matching the strongly supervised model dynamically.

  • Training used augmented 10-second patches, with pitch shifting increasing the effective training set size fivefold and auto-pool variants initialized with α = 1.
  • Models were evaluated with segment-based metrics at 1-second duration, alongside static accuracy computed by taking the maximum dynamic prediction.
  • On URBAN-SED static prediction, auto-pool achieved the highest F1 among MIL models, while the strongly supervised model was only slightly better.
  • CAP achieved dynamic F1 = 0.533 versus 0.551 for the strong model, while RAP with λ ∈ {10−3, 10−4} achieved comparable scores.
  • RAP with λ = 10−2 behaved like mean-pooling, whereas RAP with λ ≤ 10−3 and CAP tracked the strong model across classes.
  • CAP learned max-like behavior for all classes, but auto-pool still outperformed max-pooling, consistent with denser early training gradients.
  • RAP with λ = 10−3 generally localized events well on a validation clip, despite confusion between acoustically similar classes.

B. DCASE 2017 results

DCASE 2017 results favor adaptive pooling for static prediction, while soft-max pooling leads dynamically and max-pooling performs poorly. Learned pooling parameters vary by event duration, and related MedleyDB results show dataset-dependent behavior.

  • DCASE 2017: DCASE 2017 provides no strongly supervised comparison because its training data contains only clip-level annotations.
  • DCASE 2017: Auto-pool achieved the highest static F1 on DCASE 2017, while soft-max pooling achieved the highest dynamic F1 of 0.466.
  • DCASE 2017: Mean-pooling and auto-pool had comparable dynamic scores in the range of 0.41–0.45, whereas max-pooling substantially underperformed on both tasks.
  • DCASE 2017: Across DCASE classes, learned α values were larger for short events and smaller for events spanning most of the clip.
  • DCASE 2017: For long-duration DCASE classes, small α values make mean-pooling effective because bag-level and instance-level labels are similar.
  • MedleyDB: On MedleyDB, max-pooling remained weak and regularized auto-pool variants generally outperformed unconstrained auto-pool across classes.
  • MedleyDB: MedleyDB’s randomly sampled excerpts often contain instruments that are entirely active or inactive, making mean-pooling a good approximation to strong training.

VI. CONCLUSION

Across datasets, auto-pool performs best for static prediction, while regularized variants are among the strongest overall and can approach strong-model dynamic accuracy. The results also expose a trade-off: more max-like pooling favors precision over recall and may reduce dynamic performance.

  • Auto-pool consistently achieves the highest static-prediction scores across datasets.
  • Fixed pooling behavior depends on event duration and dataset characteristics, while auto-pooling often outperforms standard max-pooling.Mean-pooling suits longer events, whereas max-pooling can suit short events but may be unstable and difficult to train.
  • More max-like pooling tends to favor precision over recall in dynamic evaluation.Max-pooling needs only one instance to support bag-level detection, providing no incentive to model an event’s full duration.
  • The influence of α initialization was not empirically studied, although small initial values are recommended for gradient propagation.The recommendation is to initialize α with 0 or 1.
  • RAP models are among the best performers when λ is appropriately chosen.The paper recommends tuning λ using a strongly labeled validation set.
  • Auto-pool can nearly match dynamic prediction accuracy obtained from training with instance labels.This supports using weak labels to reduce annotation effort and cost.
Loading 1804.10070v2…