Source-linked AI summary
Learning to Separate Object Sounds by Watching Unlabeled Video
Ruohan Gao, Rogerio Feris, Kristen Grauman
TL;DR
Real-world videos mix sounds from multiple objects, while existing separation methods often depend on controlled data or clean sources. This paper learns object-level audio-visual associations from unlabeled videos with a deep MIML framework and uses the recovered bases to guide NMF separation. It reports state-of-the-art results on visually aided source separation and audio denoising, including diverse in-the-wild videos.
Problem
Realistic videos combine multiple object sounds in one audio channel, while effective existing separation methods often require multiple microphones or clean isolated sources.
Method
A deep MIML network learns associations between noisy visual object predictions and audio frequency bases, which then guide NMF source separation.
Results
The method achieves state-of-the-art results on visually aided audio source separation and audio denoising.
Takeaways & Limitations
The framework successfully separates object-level sounds from diverse videos using object-sound models learned from unlabeled video.
Takeaways & Limitations
Performance is constrained when detected objects have similar audio characteristics, are incorrectly detected, or are outside the visual field.
Abstract
from arXiv · showhide
Perceiving a scene most fully requires all the senses. Yet modeling how objects look and sound is challenging: most natural scenes and events contain multiple objects, and the audio track mixes all the sound sources together. We propose to learn audio-visual object models from unlabeled video, then exploit the visual context to perform audio source separation in novel videos. Our approach relies on a deep multi-instance multi-label learning framework to disentangle the audio frequency bases that map to individual visual objects, even without observing/hearing those objects in isolation. We show how the recovered disentangled bases can be used to guide audio source separation to obtain better-separated, object-level sounds. Our work is the first to learn audio source separation from large-scale "in the wild" videos containing multiple audio sources per video. We obtain state-of-the-art results on visually-aided audio source separation and audio denoising. Our video results: http://vision.cs.utexas.edu/projects/separating_object_sounds/
1 Introduction
The paper addresses the difficulty of separating object sounds because realistic videos mix multiple sources into one audio channel. It learns object-level audio-visual models from unlabeled video and uses visual context to guide separation.
- Motivation: Realistic videos mix sounds from multiple objects into a single audio channel, making object-level source separation difficult.Existing approaches often rely on multiple microphones or clean single-source recordings.
- Motivation: The proposed goal is to learn how different objects sound by watching and listening to unlabeled videos containing multiple sounding objects.Visual contexts provide weak supervision for discovering audio-object associations.
- Approach: The method combines image-recognition object predictions with NMF audio frequency bases, then learns which bases correspond to visible objects.A neural network is used to recover associations that are initially unknown.
- Novelty: Unlike prior work focused on low-level correlations or controlled speech and music settings, the approach learns object-level sound models from hundreds of thousands of unlabeled videos.It targets new audio-visual instances and broader in-the-wild conditions.
- Contributions: The paper introduces a deep MIML framework that injects learned acoustic-object priors into NMF source separation.The authors present this as the first study of audio source separation learned from large-scale online videos.
2 Related Work
The related work spans audio-visual localization, weakly supervised learning, multimodal representation learning, and audio source separation. The paper differs by learning object-level sound priors from broad unlabeled videos and using them to separate real audio sources.
- Audio-visual localization: Audio-visual localization methods correlate visual regions with sounds using mutual information, canonical correlation analysis, or learned associations across videos.Some early approaches assume that sounding objects are moving.
- Audio-visual representation learning: Multimodal representation-learning methods use synchronized unlabeled audio and video to improve image or audio classification, whereas this paper targets object-level source separation.The paper’s objective is to isolate sounds per object rather than improve classification.
- Audio source separation: Traditional audio separation includes ICA, sparse decomposition, CASA, NMF, probabilistic models, and deep learning, but strong performance commonly requires supervision or auxiliary cues.Isolated recordings are effective but difficult to obtain for arbitrary sources in the wild.
- Audio-visual source separation: Audio-visual separation has used motion, temporal onsets, subspace analysis, and statistical mixture models to connect visual and auditory signals.The paper instead learns how objects look and sound, whether or not they exhibit obvious motion-sound correlation.
- Audio-visual source separation: Compared with prior work centered on speakers, instruments, or single videos, this method learns a separation prior from uncurated videos containing multiple objects and audio sources.It covers broader object-level sounds, including instruments, animals, and vehicles.
- Sound generation: Unlike sound-generation methods, the paper disentangles real audio sources associated with visible objects rather than synthesizing sounds from silent video.Its goal is source separation, not audio generation.
- Weakly supervised visual learning: The approach treats automatically detected visual objects as weak labels for disentangling sounds within a mixed audio signal.These labels are inferred from video rather than manually annotated.
3 Approach
The approach learns object-level audio representations from unlabeled videos by associating NMF audio bases with weak visual object predictions. It then uses the discovered associations to guide source separation for novel videos.
- Overview: The method returns separated audio channels and the visual objects responsible for them in a new video.It learns object sounds from batches of unlabeled videos containing multiple sound sources.
- Novel-Video Separation: For a novel video, detected objects retrieve their learned bases, which are concatenated into a fixed dictionary guiding NMF source separation.The resulting basis vectors and activation scores are used to reconstruct separate audio signals for each detected object.
- Audio Basis Extraction: NMF factorizes each video’s magnitude spectrogram into M non-negative audio basis vectors and an activation matrix.The basis vectors represent spectral patterns, while the activation matrix represents their gains over time.
- Weak Supervision: Weak visual labels come from max-pooled ImageNet-trained ResNet-152 object predictions across the video’s frames.Labels above a class-probability threshold are used as video-level predictions for unlabeled training videos.
- Deep MIML Learning: The deep MIML network treats each video’s M audio bases as a bag of instances with multiple noisy object labels available only at bag level.Shared-weight branches process the bases, while pooling over sub-concepts and bases produces object predictions and an audio basis-object relation map.
- Disentangling Per-Object Bases: The learned relation map is normalized across bases to discover representative audio bases for each visual object category.The network learns audio-visual associations, and the normalized matching scores are then used to collect object-specific bases.
4 Experiments
Experiments evaluate the method on unlabeled and benchmark videos using controlled mixtures, source-separation and denoising metrics, and qualitative inspection of real multi-source videos. The method outperforms comparison systems, tolerates imperfect visual detections, and shows useful object-level audio associations, while remaining limited by similar sounds, detection errors, and incomplete audiovisual coverage.
- Datasets: The experiments use AudioSet and AV-Bench, including unlabeled training clips, 23 held-out single-source videos, and prior benchmark videos.AudioSet supplies unlabeled training data; AudioSet-SingleSource supports quantitative evaluation, while AV-Bench supports comparison with prior denoising methods.
- Evaluation: Quantitative separation tests pairwise mix single-source videos, use reserved tracks as ground truth, and report SDR across four source-pair categories.The categories are Instrument Pair, Animal Pair, Vehicle Pair, and Cross-Domain Pair; unsupervised baselines receive the best possible source matching.
- Source separation: The method significantly outperforms Visual Exemplar, Unmatched, Gaussian, and unsupervised clustering baselines while associating separated sources with acoustic objects.Both the proposed method and the unsupervised baselines use no audio-based supervision.
- Robustness: Weak labels from ImageNet-pretrained ResNet-152 perform similarly to AudioSet ground-truth labels with about 30% label corruption, indicating tolerance to imperfect visual detection.The true-label upper bound indicates how better visual models could improve results.
- Audio denoising: On AV-Bench denoising, the method outperforms the prior method on 2 of 3 videos and performs much better than two other prior audiovisual methods.The evaluation adds white noise at 0 dB SNR and reports NSDR; the prior method performs better on Guitar Solo, where hand motion may correlate with sound.
- Qualitative results: Qualitative tests on real AudioSet multi-source videos show correct piano and violin basis activations, recovery of an off-camera piano, and robustness to some inaccurate visual predictions.The results are manually inspected because these videos lack ground-truth separated sources.
- Limitations: The main failure modes occur when detected objects have similar audio characteristics or are incorrectly detected, while ImageNet coverage and camera visibility constrain the system.The paper also notes that some objects do not make sounds and some sounds originate outside the camera’s view.
5 Conclusion
The paper presents a framework that learns object sounds from unlabeled videos and uses disentangled audio bases to separate object-level sounds across diverse data and categories.
- A framework learns object sounds from thousands of unlabeled videos.
- A deep multi-instance multi-label network automatically links audio bases to object categories.
- Disentangled audio bases supervise non-negative matrix factorization to separate object-level sounds.
- The approach is effective on diverse data and object categories.
- The framework supports applications including audio event indexing, remixing, denoising for closed captioning, and instrument equalization.