Source-linked AI summary
Self-Supervised Learning of Audio-Visual Objects from Video
Triantafyllos Afouras, Andrew Owens, Joon Son Chung, Andrew Zisserman
TL;DR
The paper asks how raw video can be organized into discrete audio-visual objects despite multiple similar, moving sources and clutter. It introduces a self-supervised model that localizes and groups sound sources with attention, integrates evidence over time with optical flow, and produces embeddings for downstream speech tasks. Across these tasks, it outperforms other self-supervised approaches and is competitive with methods using supervised face detection, while also extending to non-human speakers.
Problem
Existing approaches often produce heat maps rather than discrete objects and cannot distinguish which of several same-category instances is producing a sound.
Method
LWTNet uses synchronization-based attention to detect and group sound sources, optical flow to integrate sparse evidence over time, and learned object embeddings for downstream tasks.
Results
The embeddings support four speech-oriented tasks and significantly outperform other self-supervised localization methods while achieving comparable or sometimes better performance than more strongly supervised methods.
Takeaways & Limitations
The model provides a unified raw-video representation that can be applied to other video material and fine-tuned for cartoons and puppets.
Takeaways & Limitations
For non-human speakers, lip motion is only loosely correlated with speech, suggesting less advantage from the model’s learned representation.
Abstract
from arXiv · showhide
Our objective is to transform a video into a set of discrete audio-visual objects using self-supervised learning. To this end, we introduce a model that uses attention to localize and group sound sources, and optical flow to aggregate information over time. We demonstrate the effectiveness of the audio-visual object embeddings that our model learns by using them for four downstream speech-oriented tasks: (a) multi-speaker sound source separation, (b) localizing and tracking speakers, (c) correcting misaligned audio-visual data, and (d) active speaker detection. Using our representation, these tasks can be solved entirely by training on unlabeled video, without the aid of object detectors. We also demonstrate the generality of our method by applying it to non-human speakers, including cartoons and puppets.Our model significantly outperforms other self-supervised approaches, and obtains performance competitive with methods that use supervised face detection.
1 Introduction
The paper develops a self-supervised model that converts raw video into discrete audio-visual objects by detecting, grouping, and tracking sound sources. Its learned embeddings support several speech-oriented tasks without supervised preprocessing, while transferring to non-human speakers.
- Core objective: The model transforms raw video into discrete audio-visual objects using only self-supervised audio-visual cues.The demonstrations focus on videos containing talking heads.
- Challenges: The task must distinguish among visually similar sound sources, handle motion over time, and ignore scene clutter.Multiple heads may appear simultaneously, requiring correct attribution of sound to the actual source.
- Gap: Prior self-supervised localization methods generally produce heat maps rather than discrete objects and cannot identify the sounding instance among same-category objects.The limitation applies especially when methods rely only on semantic correspondence.
- Approach: LWTNet uses synchronization cues to detect sound sources, group them into distinct instances, and track them as they move.This addresses the three challenges identified for raw video.
- Downstream tasks: The learned embeddings support source separation, talking-head localization and tracking, audio-visual alignment, and active speaker detection.These tasks previously required hand-engineered supervised pipelines.
- Results: The method significantly outperforms other self-supervised localization methods and reaches comparable or sometimes better performance than more strongly supervised methods.The tasks are learned entirely from raw audio-visual signals.
- Generality: After training on unlabeled data without preprocessing, LWTNet can be applied to other video material and fine-tuned for cartoons and puppets.The paper presents this transfer as evidence of generality beyond human speakers.
2 Related work
Prior work localized sound sources or separated audio using heat maps, correspondence cues, or supervised detection pipelines, but generally did not form and track discrete same-category instances. LWTNet combines synchronization, attention, and optical-flow-based temporal integration to produce such objects for downstream tasks.
- Sound source localization: Classical sound-source localization and segmentation used inference in simple probabilistic models, including canonical correlation analysis.
- Self-supervised localization: Synchronization-based self-supervised methods predict whether audio and video are temporally aligned, while correspondence-based methods test whether audio and a frame come from the same video.
- Limitations of prior work: Heat-map and correspondence methods generally localize likely sound regions but do not group scenes into discrete objects or distinguish among same-category instances.Without motion, correspondence models cannot identify which of several similar objects produces the sound.
- Active speaker detection: Earlier active-speaker systems used hand-engineered face detection and tracking to select candidate speakers before classification.LWTNet instead learns active speaker detection entirely from unlabeled data.
- Source separation: Prior multi-speaker separation methods either handled one on-screen speaker or relied on supervised, hand-engineered face-detection pipelines.The cited examples detect and track faces before extracting visual representations.
- Other sound-making objects: Some instrument-separation methods use semantic object detectors, while others pool visual features or produce per-pixel source maps without explicitly grouping objects.These approaches differ from discrete object formation.
- LWTNet distinction: LWTNet uses optical flow to integrate information from moving objects over time for tracking rather than merely treating flow as a motion cue.Its architecture then extracts object embeddings for downstream tasks.
3 From unlabeled video to audio-visual objects
The model converts raw video into discrete audio-visual objects by learning localized synchronization attention, integrating evidence along optical-flow tracks, and grouping peaks into instances. It trains the attention map self-supervised by distinguishing aligned audio from shifted or mismatched audio.
- Object representation: Audio-visual objects are represented as sound-source trajectories with embeddings extracted for downstream tasks.The representation includes a temporal track and attention map for each object.
- Attention estimation: The model computes fine-grained attention by comparing normalized audio and per-pixel visual embeddings at each space-time location.The resulting score is called the audio-visual attention map.
- Temporal integration: Optical flow aggregates sparse attention evidence over long-range tracks, improving robustness to camera and object motion.Dense flow is chained across frames, and attention scores are averaged along the resulting trajectories.
- Instance grouping: Distinct objects are obtained by detecting local maxima in the time-averaged synchronization map and applying non-maximum suppression.Peaks are selected greedily while suppressing nearby responses within a ρ × ρ box.
- Self-supervised learning: Self-supervised training maximizes similarity for true audio while minimizing similarity for temporally shifted or randomly mismatched audio samples.Temporal-window negatives support synchronization learning, whereas random-clip negatives produce a less motion-sensitive correspondence task.
4 Applications of audio-visual object embeddings
The learned audio-visual objects support speaker localization, multi-speaker separation, active speaker detection, and audio-video alignment without relying on supervised face detection. The same representation is used to locate speakers, condition separation, score speaking activity, and estimate synchronization offsets.
- Applications: The representation supports speaker localization, synchronization correction, active speaker detection, and multi-speaker source separation.These applications were previously often implemented with face-detection-based pipelines.
- Speaker localization: Speaker localization uses each audio-visual object’s tracked location in every frame.The tracked trajectory supplies the spatial position of a potential speaker.
- Active speaker detection: Active speaker detection thresholds the dot product between tracked visual features and the corresponding audio feature.A strong audio-visual correlation indicates that the object is speaking in that frame.
- Multi-speaker separation: Multi-speaker separation conditions a source-separation network on self-supervised audio-visual embeddings instead of face-detector features.The network isolates each speaker’s voice from a noisy mixture using the corresponding visual object.
- Audio-video alignment: Audio-video misalignment is corrected by shifting the audio and selecting the offset that maximizes frame-level audio-visual synchronization evidence.The synchronized embeddings are then recomputed for subsequent applications.
5 Experiments
Experiments evaluate the model on human and non-human speakers across localization, tracking, active speaker detection, synchronization, and source-separation settings. The results show strong performance against self-supervised, supervised, and face-detector baselines, while motion and multi-head scenes remain more challenging.
- Datasets: The evaluation uses LRS2, LRS3, and Columbia for human speech, plus Simpsons and Sesame Street footage for non-human speakers.LRS2 is used for training and evaluation, LRS3 for evaluation only, and non-human footage is annotated for active-speaker and localization labels.
- Talking head detection and tracking: The model accurately tracks speakers despite significant speaker and camera movement in continuous LRS2 scenes.The experiments include challenging motion examples, with further robustness analysis referenced in the appendix.
- Talking head detection and tracking: The model significantly outperforms other methods on talking-head localization, while optical-flow integration improves tracking beyond the No flow baseline.The No flow variant performs worse on LRS3, and enlarging the NMS window only partially alleviates the problem.
- Active speaker detection: The model outperforms previously reported active-speaker detection results on Columbia without using labeled face bounding boxes for training.Each extracted audio-visual object receives an active-speaker score, and performance is reported using per-speaker and average F1.
- Correcting misaligned visual and audio data: The self-supervised model achieves accuracy comparable to supervised synchronization methods for correcting audio-visual offsets within a ±15-frame window.An offset is counted as correct when it falls within one video frame of ground truth; SyncNet and Perfect Match are comparison methods.
- Generalization to non-human speakers: On Simpsons and Sesame Street, the model outperforms RetinaFace baselines in localization and retrieval, but performance decreases on the more challenging multi-head test sets.The multi-head results nevertheless indicate usefulness for retrieving speakers in the new domain.
6 Conclusion
The paper presents a unified raw-video model for detecting and tracking speakers, whose embeddings support multiple speech-analysis tasks without previously required supervised face detection. It identifies broader directions for extending these embeddings and framework beyond the demonstrated tasks and speech domain.
- The model learns speaker embeddings directly from raw video for downstream tasks including source separation and active speaker detection.
- The authors propose extending the embeddings to conversation diarization and face or head detection.
- The framework is presented as potentially applicable beyond speech tasks, including music and ambient-sound analysis.
A Robustness to motion.
The method remains effective on videos with large camera motion, with only a minimal performance drop for the full model. The evaluation compares high- and low-motion subsets of LRS2 and LRS3.
- The full model maintains good performance on videos with large camera motion, with a minimal performance drop.
- The robustness evaluation uses optical flow to rank videos by motion and constructs high-motion subsets from LRS2 validation and LRS3 test data.
- Table 3 breaks down talking-head detection performance across high- and low-motion subsets of LRS2 and LRS3.
B Sensitivity to NMS scale
Source-separation performance depends on the fixed NMS window: very small or large values perform worse, while an adaptive oracle provides no significant improvement.
- The robustness experiment compares multiple fixed ρ values with an oracle selecting ρ from ground-truth talking-head bounding-box size.
- Figure 9 evaluates source-separation performance on LRS2 while varying the NMS window ρ.
- Very small or large constant NMS windows reduce source-separation performance.
- Small NMS windows produce duplicate detections, whereas large windows merge separate instances.
- The adaptive oracle NMS window does not significantly improve performance over the fixed-window setting.
C Non-human speakers experiments
The authors test the method on non-human speakers using unlabeled Simpsons and Sesame Street videos, manually annotated held-out clips, and initialized models. Qualitative results show separation and enhancement capabilities, while conventional face-based baselines perform poorly.
- Training uses approximately 48 hours of The Simpsons video and 53 hours of Sesame Street video.
- Held-out test clips are manually annotated for active speakers and head bounding boxes, with no episode overlap between training and test sets.
- The test sets include both single-head localization examples and multi-head active-speaker examples.
- Separate Simpsons and Sesame Street models are initialized from the best LRS2-trained models.
- RetinaFace obtains poor average precision on the non-human-speaker videos.
- SyncNet fails out of the box because its SSD-based face-track pipeline produces virtually no correct face tracks.
- Additional Simpsons experiments train source-separation and speech-enhancement models using synthetic mix-and-separate videos.
- Qualitative results show character-voice separation, handling of same-character mixtures, and removal of background noise and music.
D Architecture details.
The paper details separate synchronization and separation architectures. The synchronization module produces audio-visual attention maps, while the separation module fuses speaker-object and mixture-audio representations.
- Architecture overview: Figure 10 presents the synchronization architecture, and Figure 11 provides the detailed separation architecture.The appendix identifies these figures as the synchronization and separation network schematics, respectively.
- Synchronization network: The synchronization network uses video and audio encoders whose embeddings share a subspace for constructing the audio-visual attention map.The architecture is a two-stream design with modality-specific encoders.
- Separation network: The separation network processes local descriptors for each speaker-object and the mixture spectrogram before modality fusion.The speaker-object descriptors are pooled from the Video Encoder’s conv6 layer.
- Separation network: A BLSTM and two fully connected layers perform modality fusion in the separation network.The listed implementation includes BLSTM and fully connected layers after the modality-specific processing modules.