Source-linked AI summary
Cooperative Learning of Audio and Video Models from Self-Supervised Synchronization
Bruno Korbar, Du Tran, Lorenzo Torresani
TL;DR
The paper asks how to learn effective audio and video representations without relying on costly manual video annotations. It trains cooperative audio and visual streams to classify temporal synchronization using carefully selected negatives and curriculum learning, achieving strong downstream audio classification and video action-recognition performance. The approach remains self-supervised and could support pretraining on video collections larger than existing labeled datasets.
Problem
Manual annotations make large-scale supervised video modeling costly, while further benchmark growth may require impractically large datasets.
Method
AVTS trains two audio-video streams to classify synchronized versus unsynchronized segments, using same-video hard negatives and curriculum learning.
Results
The learned audio features match or exceed state-of-the-art results on DCASE2014 and ESC-50, and the visual subnet improves action-recognition models on HMDB51 and UCF101.
Takeaways & Limitations
Self-supervised temporal synchronization provides general audio and visual features, with curriculum learning improving feature quality across the evaluated downstream tasks.
Takeaways & Limitations
The reported AVTS training used an established labeled video dataset for direct comparison, so the larger-scale unlabeled-data setting remains a prospective application rather than the demonstrated training setup.
Abstract
from arXiv · showhide
There is a natural correlation between the visual and auditive elements of a video. In this work we leverage this connection to learn general and effective models for both audio and video analysis from self-supervised temporal synchronization. We demonstrate that a calibrated curriculum learning scheme, a careful choice of negative examples, and the use of a contrastive loss are critical ingredients to obtain powerful multi-sensory representations from models optimized to discern temporal synchronization of audio-video pairs. Without further finetuning, the resulting audio features achieve performance superior or comparable to the state-of-the-art on established audio classification benchmarks (DCASE2014 and ESC-50). At the same time, our visual subnet provides a very effective initialization to improve the accuracy of video-based action recognition models: compared to learning from scratch, our self-supervised pretraining yields a remarkable gain of +19.9% in action recognition accuracy on UCF101 and a boost of +17.7% on HMDB51.
1 Introduction
The paper addresses the limits and annotation costs of supervised video modeling by learning audio and visual representations from self-supervised temporal synchronization. Its AVTS framework uses cooperative two-stream learning, hard negatives, and curriculum learning to improve downstream audio and video analysis.
- Self-supervised learning: AVTS learns from whether audio and video segments are temporally in sync or out of sync, without requiring manual video labels.The task is posed as binary classification over paired audio and visual inputs.
- Model design: A two-stream network processes audio and video separately, fusing the streams in later layers so they learn to work together on synchronization.This cooperative design supports subsequent use of each stream as a modality-specific feature extractor.
- Temporal synchronization: Unlike semantic correspondence methods, AVTS uses hard negatives from different temporal segments of the same video to require temporal-sensitive representations.These negatives prevent solving the task solely from semantic mismatches between unrelated videos.
- Training strategy: Because temporal synchronization is harder than semantic correspondence, the method introduces harder negatives gradually through curriculum learning.The paper reports that this strategy improves feature quality across all considered downstream tasks.
- Downstream applications: The learned audio features perform on par with or better than state-of-the-art systems on DCASE2014 and ESC-50, while the visual subnet improves action-recognition initialization.The framework also supports multisensory video categorization through finetuning of audio-visual models.
2 Technical Approach
The approach trains two modality-specific subnetworks to classify whether audio and video are temporally synchronized, using contrastive learning and curriculum-selected negatives. The architecture combines a 3D/2D video stream with a VGG-like audio stream, and the learned streams support downstream feature extraction.
- 2.1 Audio-Visual Temporal Synchronization (AVTS): AVTS classifies audio-video pairs as in-sync or out-of-sync using separately processed audio and visual features fused by a two-stream network.The audio and visual inputs span the same temporal duration, and the fusion function addresses synchronization.
- 2.2 Choice of Loss Function: Contrastive loss replaces cross-entropy during representation learning because it provides more consistent and robust optimization from scratch.At convergence, synchronization is predicted by thresholding the distance between audio and visual feature representations.
- 2.3 Selection of Negative Examples: Hard negatives pair samples from different temporal slices of the same video, while easy negatives pair audio and video from different videos.Hard negatives target temporal synchronization rather than semantic correspondence; super-hard negatives use overlapping audio and visual sequences.
- 2.4 Curriculum Learning: Introducing hard negatives only after initial training on easy negatives makes the objective easier to optimize and improves downstream feature quality.The hard-negative stage is introduced between the 40th and 50th epoch.
- 2.5 Architecture Design: The video stream uses an MCx architecture with early 3D convolutions followed by 2D convolutions, while the audio stream uses the VGG model of Chung and Zisserman.The MCx design uses early temporal modeling and later spatial processing; the audio input is converted to MP3 and represented with FFT filterbank features.
3 Experiments
Experiments evaluate AVTS synchronization, audio representations, action recognition pretraining, and multimodal recognition. Results show curriculum learning and self-supervised pretraining improve performance across these tasks.
- AVTS evaluation: 9.4% higher AVTS accuracy results from introducing a 75/25% easy/hard-negative curriculum after initial training on easy negatives.Including hard negatives in the first stage is deleterious, while super-hard negatives make optimization overly difficult.
- AVTS evaluation: 78% versus 74% on Kinetics and 86% versus 81% on AudioSet show higher AVTS accuracy than L3-Net.Hard negatives help AVTS maintain 70% accuracy on Kinetics when hard negatives appear at test time.
- Action recognition: 16.7% and 13.0% gains improve MC3 action recognition on UCF101 and HMDB51 after AVTS pretraining versus learning from scratch.Using AudioSet for AVTS pretraining further reaches 89.0% on UCF101, 1.5% below full Kinetics supervision.
- Audio classification: AVTS audio features generalize extremely well on ESC-50 and DCASE2014, achieving performance superior or close to the state of the art.The AVTS procedure, rather than the architecture alone, accounts for the effectiveness because the same audio subnet performs poorly when trained directly on the benchmarks.
- Multimodal recognition: 87.0% versus 82.1% shows higher UCF101 multisensory action-recognition accuracy than the concurrent method of Owens and Efros.The comparison also favors AVTS in video-only recognition: 85.8% versus 77.6%.
- Cross-task findings: Curriculum learning benefits AVTS, audio classification, and action recognition, with comparisons conducted against L3-Net using matched data amounts.The study uses contrastive-loss AVTS training and evaluates audio conv5 features without further finetuning.
4 Related work
Prior work uses cross-modal correlations for self-supervised representation learning, while this paper extends the setting from single frames and semantic correspondence to video clips and temporal synchronization.
- Cross-modal self-supervision: Earlier methods used audio to pretrain visual models or visual predictions to supervise audio and depth-image models.
- Audio-visual correspondence: Audio-Visual Correspondence predicts whether an audio-video pair is in true correspondence to jointly learn audio and visual representations.
- Distinction from prior work: Unlike single-frame approaches focused on semantics, this method uses video clips to learn spatiotemporal features.
- Scope: The paper broadens audio-visual synchronization from mouth-motion and speech correlation to arbitrary human activities and general feature learning.
5 Conclusions
The paper concludes that AVTS learns effective audio and visual models through cooperative synchronization training. Curriculum learning and harder negatives improve downstream representations, while the self-supervised setup supports larger unlabeled video collections.
- Conclusions: AVTS learns general audio and visual models by having separate streams cooperate toward audio-visual synchronization classification.
- Conclusions: Video clips and out-of-sync pairs make the model address temporal alignment rather than merely semantic correspondence.
- Conclusions: Curriculum learning, which introduces harder negatives in a second training stage, significantly improves feature quality on all end tasks.
- Conclusions: Because AVTS requires no manual labeling, it can be applied to video collections much larger and more diverse than existing labeled datasets.The authors present this as a possibility for improving model generality and narrowing the gap with fully supervised pretraining.
A.1 Performance on downstream tasks as a function of AVTS training epochs
ESC-50 accuracy from AVTS-trained conv5 audio features increases substantially with training compared with randomly initialized features.
- Performance over training: 80.6% versus 45.2% shows over 30% improvement in ESC-50 accuracy over randomly initialized features.Accuracy is measured using a linear SVM on conv5 audio features after AVTS training on the full AudioSet dataset.
A.2 Performance on downstream tasks as a function of number of AVTS examples
The paper examines how AVTS training-set size affects audio scene classification, using AudioSet subsets and comparison with full-Kinetics pretraining.
- Table 3 compares audio scene-classification results across AVTS pretraining on Kinetics, AudioSet, and SoundNet.
- A size-matched AudioSet subset contains 230K videos, corresponding to randomly sampled 13% of each original AudioSet class.
- Table 5 varies AVTS training-set size with AudioSet subsets and compares accuracy against audio features trained on the full Kinetics dataset.
A.3 Cross-modal sound localization
The model uses audio gradients to localize sound within video frames through Grad-CAM, with visualizations filtering low activations.
- Sound localization is obtained by back-propagating audio gradient activations to video frames using Grad-CAM.
- Figure 4 visualizes the resulting sound localizations after filtering low activations for clearer display.