Source-linked AI summary
Co-Separating Sounds of Visual Objects
Ruohan Gao, Kristen Grauman
TL;DR
Object sounds overlap in single-channel video, and existing artificial-mixing training requires impractical solo recordings while potentially missing real source correlations. The paper introduces co-separation, which learns object-level separation from unlabeled multi-source videos through consistent identification of sounds for similar-looking objects. It reports state-of-the-art visually guided separation and denoising across MUSIC, AudioSet, and AV-Bench.
Problem
Single-channel videos mix sounds from multiple objects, while existing mix-and-separate training depends on clean solo recordings and may not capture correlations among real sources.
Method
Co-separation trains on paired unlabeled multi-source videos, enforcing object-level consistency for separated tracks associated with similar-looking objects.
Results
The approach achieves state-of-the-art results on visually guided audio source separation and audio denoising across the MUSIC, AudioSet, and AV-Bench datasets.
Takeaways & Limitations
The method can separate an object’s sound in a test video even when that object was never observed individually during training.
Takeaways & Limitations
The paper notes that SAR measures artifacts rather than separation accuracy, so high SAR can occur without well-separated signals.
Abstract
from arXiv · showhide
Learning how objects sound from video is challenging, since they often heavily overlap in a single audio channel. Current methods for visually-guided audio source separation sidestep the issue by training with artificially mixed video clips, but this puts unwieldy restrictions on training data collection and may even prevent learning the properties of "true" mixed sounds. We introduce a co-separation training paradigm that permits learning object-level sounds from unlabeled multi-source videos. Our novel training objective requires that the deep neural network's separated audio for similar-looking objects be consistently identifiable, while simultaneously reproducing accurate video-level audio tracks for each source training pair. Our approach disentangles sounds in realistic test videos, even in cases where an object was not observed individually during training. We obtain state-of-the-art results on visually-guided audio source separation and audio denoising for the MUSIC, AudioSet, and AV-Bench datasets.
1. Introduction
Real-world videos mix sounds from coexisting objects in a single audio channel, while mix-and-separate training relies on impractical solo recordings and may miss source correlations. Co-separation instead learns object-level sound separation from naturally occurring multi-source videos by enforcing consistent identification across similar objects.
- Objects’ sounds are intertwined in single-channel recordings, making object-level separation practically important for denoising, indexing, equalization, remixing, and dialog following.
- Mix-and-separate methods create artificial mixtures and aim to recover the original unmixed signals.
- Collecting enough clean solo recordings is impractical and difficult to scale beyond classes such as human speakers and musical instruments.
- Artificially mixed clips assume independent sources, so correlations between real sound sources may go uncaptured.
- Co-separation trains on pairs of naturally occurring multi-source videos, requiring separated tracks to remain consistently identifiable at the object level.
- At test time, the method separates sounds from an individual novel video, despite co-separation operating only during training.
- Co-separation benefits from in-the-wild sound mixtures, natural source correlations, and partnerships between similarly appearing objects across videos.
- The approach achieves state-of-the-art separation and denoising results across three benchmark datasets and has its largest margin over prior methods and baselines on noisy AudioSet videos.
2. Related Work
Prior audio-visual separation methods largely use artificially mixed training videos, while related approaches differ in how they model or localize sound. This work separates existing multi-object audio using visual object detections rather than synthesizing sounds or merely localizing their sources.
- Audio-only separation methods include multi-microphone, blind single-channel, and deep-learning approaches, with mix-and-separate training used to create artificial examples.
- Recent audio-visual methods separate speech, musical instruments, and other objects, but almost all train by artificially mixing video clips.
- Compared with AV-MIML and Sound of Pixels, the approach uses localized object detections and end-to-end learning to guide separation.
- Sound-localization methods identify where audio originates in video frames but do not separate multiple objects from a single-channel signal.
- Unlike sound-generation methods, this work separates an existing real audio track rather than synthesizing plausible new sounds.
3. Approach
The approach learns object-level audio-visual separation from unlabeled multi-source video by combining video-level reconstruction with within-video object separation and object consistency. A detector supplies localized objects, while an end-to-end network predicts object-conditioned spectrogram masks for mixed audio.
- Co-Separation Framework: The framework detects objects in paired videos, mixes their audio tracks, and separates sounds for each detected visual object.The paired-video pipeline uses video-level audio mixing while conditioning separation on localized object regions.
- Co-Separation Framework: Within-video object-level separation extends mix-and-separate training to multi-source videos and captures correlations between sound sources.The method can learn object sound models from noisy multi-source training videos rather than requiring isolated examples.
- Network Architecture: The separator takes a mixed audio signal and detected object region as input, jointly analyzes audio and visual features, and outputs the sound associated with that object.Visual features from a ResNet-18 branch are combined with spectrogram features in a U-NET-style audio branch.
- Training Objective: A co-separation loss encourages separated object sounds to reconstruct each source video's audio, using magnitude-spectrogram relationships and per-pixel L1 mask supervision.The reconstruction relationship is approximate in the frequency domain because of phase interference; training therefore applies the loss over predicted masks.
- Training Objective: An object-consistency loss makes the predicted spectrogram category agree with the weak label of its input visual object.The auxiliary audio classifier discovers sound classes during training without pretrained sound models or single-source audio examples.
- Training and Inference: The total training objective combines co-separation mask loss with weighted object consistency, and inference processes videos using overlapping sliding windows.The method assumes every detected object in a 10-second clip may be a sound source, including objects that are silent in some or all frames.
4. Experiments
Experiments evaluate co-separation across MUSIC, AudioSet, and AV-Bench using curated and in-the-wild videos, source separation, denoising, unseen-object tests, and embedding analysis. Across these settings, the method improves separation and denoising, learns from multi-source videos, handles objects absent as individual training examples, and produces object-characteristic sound embeddings.
- Evaluation setup: Experiments cover MUSIC, AudioSet, and AV-Bench, including source separation, visually assisted denoising, and qualitative analysis.MUSIC and AudioSet evaluate separation, while AV-Bench evaluates denoising; real multi-source videos are also used qualitatively.
- Source separation: On MUSIC, co-separation consistently outperforms baselines on SDR and SIR and improves when trained with both solo and duet videos.The method achieves its best performance when training combines single-source and multi-source videos, unlike the SoP baseline.
- Source separation: On AudioSet, training on unlabeled multi-source videos outperforms prior methods and baselines by a large margin on the challenging evaluation.The result supports learning from noisy, less curated videos containing many multi-source examples.
- Generalization: The method separates objects never observed individually during training, performing consistently across violin, saxophone, and guitar test combinations.In the toy experiment, guitar solos are absent from training, yet the model learns from duet videos containing guitar and other sounds.
- Audio denoising: On AV-Bench, the method achieves state-of-the-art denoising performance on 2 of 3 videos despite learning only from unlabeled video.The comparison notes that a competing method’s use of motion in manually segmented regions may help on Guitar Solo.
- Qualitative analysis: Separated sounds cluster by object category in t-SNE embeddings, while visual retrieval results generally align detected objects with their corresponding audio.Failure cases commonly involve visually similar objects, such as clothing stripes and harp shadows.
5. Conclusion
The paper presents an object-level audio-visual source separation framework that uses noisy object detections to learn from large-scale unlabeled videos. CO-SEPARATION achieves state-of-the-art results on visually guided source separation and audio denoising.
- CO-SEPARATION associates localized object regions in videos with their characteristic sounds.
- Noisy object detections provide supervision for learning from large-scale unlabeled videos.
- The framework achieves state-of-the-art results on visually guided audio source separation and audio denoising.
- Future work will explore spatio-temporal object proposals and object motion, especially for sounds with similar frequencies.