PaperlayerView exact revision on arXiv

Source-linked AI summary

Localizing Visual Sounds the Hard Way

Honglie Chen, Weidi Xie, Triantafyllos Afouras, Arsha Nagrani, Andrea Vedaldi, Andrew Zisserman

arXiv:2104.02691v1cs.CVeess.ASeess.IV

TL;DR

The paper addresses unsupervised localization of visible sound sources in videos, where prior methods may poorly cover the sounding object. It mines hard background regions for contrastive learning using differentiable thresholding and Tri-maps, achieving state-of-the-art performance while introducing the larger VGG-SS benchmark.

  • Problem

    The task is to localize visible sound-emitting objects without manual annotations, while prior models may detect too little or too much of the object.

  • Method

    The method automatically mines low-correlation image regions as hard negatives for contrastive learning using differentiable thresholding and Tri-maps to ignore uncertain regions.

  • Results

    The algorithm achieves state-of-the-art performance on Flickr SoundNet and VGG-SS, while VGG-SS provides annotations for more than 5K videos across 200 categories.

  • Takeaways & Limitations

    Explicitly considering hard negative image locations significantly boosts existing sound-source localization methods, and VGG-SS supplies a larger video-based benchmark for evaluation.

  • Takeaways & Limitations

    The approach is evaluated in an unsupervised setting that replaces an oracle assuming ground-truth emitting regions for each visual-audio pair.

Abstract

from arXiv · show

The objective of this work is to localize sound sources that are visible in a video without using manual annotations. Our key technical contribution is to show that, by training the network to explicitly discriminate challenging image fragments, even for images that do contain the object emitting the sound, we can significantly boost the localization performance. We do so elegantly by introducing a mechanism to mine hard samples and add them to a contrastive learning formulation automatically. We show that our algorithm achieves state-of-the-art performance on the popular Flickr SoundNet dataset. Furthermore, we introduce the VGG-Sound Source (VGG-SS) benchmark, a new set of annotations for the recently-introduced VGG-Sound dataset, where the sound sources visible in each video clip are explicitly marked with bounding box annotations. This dataset is 20 times larger than analogous existing ones, contains 5K videos spanning over 200 categories, and, differently from Flickr SoundNet, is video-based. On VGG-SS, we also show that our algorithm achieves state-of-the-art performance against several baselines.

1. Introduction

The paper targets unsupervised localization of visible sound sources, addressing prior methods’ limited object coverage. It introduces automatic hard-background mining with Tri-map contrastive learning and a larger annotated video benchmark.

  • The task is to localize visual objects emitting characteristic sounds in videos by learning correlations between visual and audio streams.
  • Prior localization models can detect too little or too much of the sounding object because their spatial attention does not ensure good object coverage.
  • The proposed training scheme explicitly searches for background regions that are easily confused with the sounding object.
  • Low-correlation regions are automatically mined as contrastive negatives through differentiable thresholding, while Tri-maps provide an ignore zone for uncertain regions.
  • VGG-Sound Source provides bounding-box annotations for more than 5K videos across 200 categories, making it 20× larger and more diverse than existing sound-localization benchmarks.

2. Related Work

Related work progresses from shallow probabilistic and correlation-based methods to deep audiovisual models and source-separation objectives. Existing benchmarks differ in scale and annotation type, with several providing only temporal localization.

  • Early methods used shallow probabilistic models or spatio-temporal tubes linked to audio through canonical correlation analysis.
  • Modern approaches typically use dual-stream networks and contrastive learning to match audio and visual representations from the same video.
  • Alternative methods learn audiovisual source separation, sometimes incorporating motion, but commonly rely on videos containing single sound sources.
  • Flickr SoundNet is a standard spatial sound-source benchmark built from annotated single frames randomly sampled from videos.
  • AVE contains 4,143 ten-second clips across 28 categories, whereas LLP contains 11,849 clips across 25 categories and provides frame-level test labels.
  • VGG-SS is compared with existing benchmarks by image and class counts, while AVE and LLP are marked as temporal-only rather than spatial localization datasets.

3. Method

The method localizes sounding objects by matching audio with spatial visual features and automatically mining positive, negative, and uncertain regions for contrastive learning. It replaces unavailable masks with differentiable thresholding and Tri-maps while retaining hard negatives from image backgrounds.

  • Audio-Visual Feature Representation: A dual-stream CNN extracts audio and visual representations, whose cosine similarity produces a spatial response map for localization.The visual representation retains spatial extent, allowing audio responses to be evaluated at different image locations.
  • Audio-Visual Localization with an Oracle: Contrastive learning combines positive responses from sounding regions with hard negatives from background regions and easy negatives from other images.The hard-negative term explicitly penalizes background locations within images that contain the sounding object.
  • Audio-Visual Localization with an Oracle: Prior self-supervised localization methods generally use cross-image negatives but omit hard negatives from nonsounding regions within the same image.The paper identifies this omission as important because it can lead to imprecise sound-source coverage.
  • Self-supervised Audio-Visual Localization: Without manual masks, the method computes a correspondence map and thresholds it to obtain pseudo-ground-truth positive and negative regions.The self-supervised procedure treats model predictions as pseudo-ground-truth for retraining.
  • Self-supervised Audio-Visual Localization: Differentiable thresholding replaces the nondifferentiable Heaviside function, with a temperature parameter controlling threshold sharpness.The smoothed operator preserves end-to-end differentiability.
  • Self-supervised Audio-Visual Localization: Applying separate positive and negative thresholds creates a Tri-map whose intermediate uncertain region is ignored during training.For example, scores above 0.6 can be positive, below 0.4 negative, and scores in [0.4, 0.6] uncertain.

4. The VGG-Sound Source Benchmark

VGG-Sound Source (VGG-SS) addresses limitations of Flickr SoundNet by providing a substantially larger, video-based benchmark with bounding-box annotations for sounding objects. Its semi-automatic pipeline combines detector-generated candidates, manual annotation, and video verification to produce a cleaned dataset.

  • Benchmark construction: VGG-SS provides bounding-box annotations for sounding objects in over 5K videos spanning roughly 220 classes.The benchmark is built from VGG-Sound and is intended for audio-visual localization evaluation.
  • Annotation pipeline: The pipeline first uses Faster R-CNN and semantic word matching to generate bounding-box candidates for the 15K-clip VGG-Sound test set.This automatic stage produces roughly 8K annotated frames.
  • Annotation pipeline: Manual annotation removes difficult or unclear categories, marks interaction regions, and handles multiple instances according to separability and instance count.Approximately 50 challenging environmental-sound classes are removed; separable groups with fewer than five instances are annotated separately.
  • Annotation pipeline: Manual verification watches five-second video windows to confirm that the annotated object corresponds to the sound, especially when multiple candidate instances are visible.This final check helps resolve cases where only one of several similar objects produces the sound.
  • Final dataset: The staged process reduces the initial 309 classes and 15K frames to roughly 260 classes and 8K frames, yielding a cleaned large-scale benchmark.The final dataset is described as high quality and 20 times larger than the existing benchmark.
  • Benchmark motivation: 20× larger than the existing benchmark, VGG-SS improves on Flickr SoundNet’s limited instances, categories, still-frame format, and missing object annotations.Flickr SoundNet contains about 250 instances and approximately 50 categories, with only selected reference frames.

5. Experiments

Experiments train and evaluate audio-visual localization models on Flickr SoundNet and VGG-Sound using matched image-audio inputs. Evaluation reports cIoU and AUC on the Flickr SoundNet and VGG-SS test sets under varying training-set sizes.

  • Datasets: Training uses Flickr SoundNet and VGG-Sound, with center frames from raw videos used as visual inputs.Additional video frames were tested for training without considerable performance change.
  • Datasets: Flickr SoundNet experiments follow established splits with self-supervised training on 10K or 144K image-audio pairs.The dataset contains over 2 million unconstrained Flickr videos.
  • Datasets: VGG-Sound experiments use training sets of 10K, 144K, or the full collection of image-audio pairs.VGG-Sound contains over 200K clips across 309 sound categories.
  • Evaluation: Models are evaluated with Consensus Intersection over Union (cIoU) and Area Under Curve (AUC) on two test sets.The test sets are Flickr SoundNet and the proposed VGG-Sound Source benchmark.
  • Input construction: For VGG-Sound training, a middle video frame and a 3s surrounding audio segment are converted into an equivalent image-audio pair.This aligns video-based clips with the image-audio training format.
  • Implementation details: The implementation uses ResNet18 backbones for both streams, ImageNet visual pretraining, and a 10^-4 learning rate with batch size 256.The reported setup also uses Adam optimization and evaluates with full-length audio spectrograms.

6. Results

The method outperforms prior approaches on Flickr SoundNet and VGG-SS, while ablations show that Tri-map handling of uncertain regions is especially important. Qualitative results also indicate cleaner localization maps and generalization to unseen categories.

  • Comparison on Flickr SoundNet: The method outperforms prior approaches on Flickr SoundNet across training sets and data amounts.With 10k Flickr SoundNet samples, it reports 0.546% vs. 0.582%; using 144k samples yields a further 5% improvement over the previous state of the art.
  • Ablation Analysis: Tri-map training improves performance by around 2–4% over treating complementary regions uniformly as negatives.The gain reflects leaving uncertain regions ignored so the model can self-tune.
  • Ablation Analysis: The selected hyperparameters provide a strong balance between positive and negative regions, with model-e using ϵp = 0.65 and ϵn = 0.4.The model is generally robust to different hyperparameter settings.
  • Comparison on VGG-Sound Source: On VGG-SS, the method outperforms all baselines by around 5%, although cIoU is lower for every model than on Flickr SoundNet.The lower scores indicate that VGG-SS is more diverse and challenging.
  • Qualitative Results: The method produces cleaner heatmaps, with hard negatives penalized during training and low-correlation regions highlighted as negatives.Tri-map thresholding identifies positive objects, background negatives, and an uncertain region for training.
  • Open Set Audio-visual Localization: Performance differs by only 2% between heard and unheard evaluations, demonstrating generalization to categories not seen during training.The authors attribute this to semantic similarities between some audio-visual categories, such as speech and singing.

7. Conclusion

The paper revisits unsupervised visual sound source localization with a larger, more challenging benchmark and automatic hard-negative mining. Combining differentiable thresholding with Tri-maps substantially outperforms the state of the art.

  • Conclusion: The proposed technique explicitly mines hard-negative image locations within images containing the sounding object.Differentiable thresholding and Tri-maps implement this idea in a self-supervised contrastive framework.
  • Conclusion: The method significantly outperforms the state of the art on unsupervised visual sound source localization.

A. Evaluation metric

The evaluation uses consensus intersection over union and area under the success-ratio curve to assess predicted sound-source masks against human-annotated ground truth.

  • cIoU: Consensus intersection over union compares predicted masks with a weighted ground-truth mask built from multiple human bounding-box annotations.The Flickr SoundNet dataset provides three annotations per image, and agreement requires C = 2 opinions.
  • cIoU: cIoU thresholds prediction-map values at τ = 0.5 before comparing positive pixels with the ground-truth region.A(τ) contains pixels whose prediction exceeds τ, while G contains pixels with positive ground-truth values.
  • Examples: Figure 6 provides example predictions together with their calculated cIoU values.
  • AUC: AUC integrates the success ratio over cIoU thresholds from 0 to 1.The success ratio is the fraction of samples whose cIoU exceeds each threshold τ2.

A.1. Tri-map visualisation

The appendix presents additional Tri-map visualizations and dataset examples, extending the visual examples beyond the main video results.

  • Tri-map visualisation: Figure 7 shows additional image-based Tri-map results alongside the video examples.
  • Dataset Examples: The appendix also includes more dataset examples and the full 220-class VGG-SS list.It additionally identifies classes removed from the original VGG-Sound dataset.

B.1. VGG-SS annotation interface

The VGG-SS annotation workflow uses a manual interface to verify audio clips and supports bounding-box-based dataset organization and class-frequency reporting.

  • Annotation interface: LISA presents 5-second clips for annotation, with “Yes,” “No,” and “Not Sure” choices based on whether the target sound is heard.The interface also includes controls for playback and recentering the key frame to annotate.
  • Bounding-box organization: Examples are organized by the number of annotated bounding boxes: one-box examples occupy four rows, two-box examples two rows, and multi-box examples the final row.The sampling procedure randomly selects images from these three bounding-box groups.
  • Class statistics: The VGG-SS test set is summarized with a per-class frequency bar chart and a corresponding indexed list of class names and clip counts.Reported classes include telephone bell ringing with 40 clips, driving snowmobile with 39, and playing violin or fiddle with 38.
  • Class coverage: The class list spans diverse sound sources, including instruments, vehicles, animals, speech, household devices, and environmental sounds.Examples include playing tambourine, railroad car or train wagon, cat caterwauling, people eating noodle, and people sniggering.
Loading 2104.02691v1…