Source-linked AI summary
VGGSound: A Large-scale Audio-Visual Dataset
Honglie Chen, Weidi Xie, Andrea Vedaldi, Andrew Zisserman
TL;DR
The paper addresses scalable collection of large-scale audio-visual data from unconstrained media while maintaining correspondence and low label noise. It proposes a computer-vision-based filtering pipeline and uses it to curate VGG-Sound and establish audio-recognition baselines. The resulting dataset contains over 200k videos across 309 classes and supports audio and audio-visual research.
Problem
The paper seeks a scalable way to collect large-scale audio data from unconstrained open-source media while maintaining audio-visual correspondence and low label noise.
Method
The authors build a cascade that obtains videos, uses image classification to identify visually corresponding sound sources, filters audio, and iteratively curates the dataset with limited manual input.
Results
VGG-Sound contains over 200k videos across 309 audio classes, and the paper establishes audio-recognition baselines using different CNN architectures and aggregation approaches.
Takeaways & Limitations
VGG-Sound is intended for audio recognition and audio-visual prediction tasks, including sound localization and object-specific sound separation.
Takeaways & Limitations
The curation process assumes noisy samples have no systematic bias, model errors differ across ensemble members, and sound-producing objects exhibit informative visual patterns.
Abstract
from arXiv · showhide
Our goal is to collect a large-scale audio-visual dataset with low label noise from videos in the wild using computer vision techniques. The resulting dataset can be used for training and evaluating audio recognition models. We make three contributions. First, we propose a scalable pipeline based on computer vision techniques to create an audio dataset from open-source media. Our pipeline involves obtaining videos from YouTube; using image classification algorithms to localize audio-visual correspondence; and filtering out ambient noise using audio verification. Second, we use this pipeline to curate the VGGSound dataset consisting of more than 210k videos for 310 audio classes. Third, we investigate various Convolutional Neural Network~(CNN) architectures and aggregation approaches to establish audio recognition baselines for our new dataset. Compared to existing audio datasets, VGGSound ensures audio-visual correspondence and is collected under unconstrained conditions. Code and the dataset are available at http://www.robots.ox.ac.uk/~vgg/data/vggsound/
1. INTRODUCTION
The paper targets scalable collection of large-scale audio-visual data from unconstrained media while preserving audio-visual correspondence and low label noise. It introduces an automated pipeline, curates VGG-Sound, and establishes audio-recognition baselines.
- Large-scale audio datasets often require extensive human effort, limiting scalability or domain breadth.
- The proposed computer-vision pipeline collects unconstrained open-source audio data with visually evident sound sources and low label noise using minimal manual effort.
- VGG-Sound contains over 200k YouTube video clips across 309 audio classes, with at least 200 clips per class and corresponding visual frames and audio.
- The paper establishes audio-recognition baselines by investigating CNN architectures and aggregation methods, including global average pooling and NetVLAD.
- VGG-Sound is intended for audio recognition and audio-visual tasks such as sound localization and object-specific sound separation.
2. RELATED WORK
Prior audio datasets range from small, relatively clean collections to large unconstrained resources, but scale, manual curation, or audio-visual correspondence remain differing design constraints. The paper positions VGG-Sound within this landscape and the shift toward deep audio recognition.
- UrbanSound, Mivia, and DCASE-related datasets provide relatively clean audio data but are often too small for data-hungry deep neural networks.
- AudioSet provides more than 2 million YouTube clips for audio and audio-visual research but required extensive human rating and filtering.
- Existing manually curated resources differ in whether videos are available and whether the sound source is visually evident, as summarized by the dataset statistics table.
- Traditional audio recognition used GMMs or SVMs with handcrafted features, while more recent approaches use CNNs or RNNs.
3. THE VGG-SOUND DATASET
VGG-Sound is built through a scalable cascade that generates candidate classes and videos, verifies visual and acoustic correspondence, filters noise, and iteratively expands the retained dataset. The resulting collection contains over 200k videos across 309 sound classes.
- Dataset pipeline: The generation process progressively filters candidate classes and videos into reliably annotated clips, with manual input limited to a few well-defined tasks.
- Dataset pipeline: The class list favors real-world sounds that can be visually grounded and verified, while VGG-Sound uses one flat label per clip.
- Dataset pipeline: The pipeline obtains candidate videos through YouTube queries, including transformed, translated, and synonymous query variants.
- Dataset pipeline: Semantic word embeddings match sound classes to OpenImage visual classes when direct label matching fails, producing visual signatures for video-frame selection.
- Dataset pipeline: The final filtering procedure combines visual and audio checks, manual sampling, classifier predictions, and assumptions about model ensembling and sound-producing visual patterns.
- Dataset outcome: VGG-Sound contains over 200k videos in 309 classes, with each class containing 200–1000 audio-visual corresponding clips after deduplication.
4. EXPERIMENTS
The experiments establish audio-recognition baselines on VGG-Sound using standard metrics and compare model architectures and test sets. Deeper architectures perform better, while the full VGG-Sound test set is easier than the AudioSet-overlap test set.
- Experimental setup: The study evaluates audio recognition on VGG-Sound and on AStest, an AudioSet–VGG-Sound intersection containing 164 classes and 7k clips.Models are trained on VGG-Sound and tested on both AStest and the VGG-Sound test set.
- Evaluation metrics: Evaluation uses mean average precision, AUC, and equivalent d-prime class separation.
- Implementation details: Training samples 5s audio segments converted into 257×500 spectrograms, whereas testing feeds the full 10s audio clip to the network.
- Implementation details: The baselines use CNN architectures trained with Adam and cross-entropy loss, with a softmax output for the flat class labels.
- Results: Deeper architectures clearly outperform the shallow architecture in comparisons among Model-A, Model-B, and Model-C.
- Results: The full VGG-Sound test set outperforms AStest because AStest contains harder clips with multiple sounds, although the Top5-accuracy gap is smaller.
5. CONCLUSION
The paper concludes by presenting VGG-Sound as a large-scale audio-visual dataset collected through an automated pipeline for unconstrained conditions, and by providing CNN audio-recognition baselines.
- Conclusion: VGG-Sound contains more than 200k videos and 309 classes for audio recognition under unconstrained conditions.
- Conclusion: The paper compares CNN architectures to establish baseline results for audio recognition on VGG-Sound.
A. LIST OF VGG-SOUND CLASSES
The VGG-Sound class list spans diverse real-world sound categories, including vehicles, speech, music, animals, nature, tools, and household or activity sounds.
- People: Human vocal classes include female and male speech, singing, laughter, crying, cheering, whispering, coughing, and sneezing.
- Music: Music classes cover orchestras, singing, and instruments such as guitars, piano, violin, flute, drums, horns, and synthesizer.
- Activities and objects: Activity and object classes include typing, printing, vacuuming, mowing, hammering, doors, drawers, water sounds, sports, and household actions.
- Nature: Nature and environmental classes include wind, rain, thunder, waterfalls, ocean sounds, fire, tornadoes, and volcanoes.