Source-linked AI summary
Audio-Visual Instance Discrimination with Cross-Modal Agreement
Pedro Morgado, Nuno Vasconcelos, Ishan Misra
TL;DR
The paper addresses how to learn useful audio-visual representations from naturally paired video and audio beyond single-instance discrimination. It introduces contrastive cross-modal discrimination and Cross-Modal Agreement, reporting improved representations and downstream performance, while identifying limitations of instance discrimination.
Problem
Single-instance discrimination misses interactions between instances, can treat semantically related samples as negatives, and does not directly optimize within-modality visual similarity.
Method
AVID contrasts video representations against multiple audios and vice versa, while CMA groups videos similar in both video and audio spaces to optimize within-modal similarity.
Results
CMA improves upon AVID on action recognition tasks across Kinetics, UCF-101, and HMDB-51 under linear probing and full fine-tuning, while Cross-AVID outperforms prior audio-visual SSL methods in reported settings.
Takeaways & Limitations
Cross-modal discrimination and cross-modal agreement provide useful signals for learning improved visual and audio representations in a multimodal setting.
Abstract
from arXiv · showhide
We present a self-supervised learning approach to learn audio-visual representations from video and audio. Our method uses contrastive learning for cross-modal discrimination of video from audio and vice-versa. We show that optimizing for cross-modal discrimination, rather than within-modal discrimination, is important to learn good representations from video and audio. With this simple but powerful insight, our method achieves highly competitive performance when finetuned on action recognition tasks. Furthermore, while recent work in contrastive learning defines positive and negative samples as individual instances, we generalize this definition by exploring cross-modal agreement. We group together multiple instances as positives by measuring their similarity in both the video and audio feature spaces. Cross-modal agreement creates better positive and negative sets, which allows us to calibrate visual similarities by seeking within-modal discrimination of positive instances, and achieve significant gains on downstream tasks.
1. Introduction
The paper uses naturally co-occurring audio to learn audio-visual representations through contrastive cross-modal discrimination, then extends this framework to group semantically related videos. It reports improved downstream action-recognition performance from these design choices.
- Motivation: Freely occurring audio can connect visual scenes that go together, providing self-supervised learning signal for video representations.A single sound may evoke multiple related scenes while separating unrelated ones.
- Prior approach: Existing audio-visual methods commonly learn from binary verification of whether a video and audio pair is correct or incorrect.Incorrect pairs can use out-of-sync audio or audio from a different video.
- Problem: Single-pair verification misses the opportunity to reason about the broader data distribution across multiple instances.
- AVID: AVID contrasts each video representation against multiple audios, and vice versa, to learn a cross-modal similarity metric.The method groups video and audio instances that co-occur within a contrastive learning framework.
- AVID: Cross-modal discrimination is more powerful than within-modal discrimination for learning representations from video and audio.The comparison concerns predicting which audio matches a video versus identifying clips from the same video.
- CMA: CMA groups videos similar in both video and audio spaces, enabling within-modal visual optimization and improvements over AVID on Kinetics, UCF-101, and HMDB-51.The gains are reported under both linear probing and full fine-tuning evaluation protocols.
2. Related work
Related work spans self-supervised representation learning, audio-visual correspondence, contrastive learning, and multi-view learning. The paper positions AVID and CMA as using naturally aligned sound and video to contrast multiple samples and model cross-modal agreement.
- Self-supervised learning: Self-supervised methods learn transferable representations by reconstructing inputs or imposing representation constraints, often using structural properties of visual data as pretext tasks.
- Self-supervised learning: Naturally available alignment between video and audio provides an unsupervised learning signal for representation learning.
- Audio-visual correspondence: Audio-visual correspondence typically aligns visual and audio inputs with binary classification, but most methods process only one video-audio pair at a time.
- Audio-visual correspondence: The paper uses contrastive loss to oppose many samples simultaneously and reports better performance than recent methods using audio-visual correspondence.
- Contrastive learning: Unlike standard instance discrimination, CMA models groups of related videos and enables within-modal similarity optimization using cross-modal agreement.
- Multi-view learning: Multi-view learning seeks common representations from multiple views, while sound and video provide a multi-view source of supervision in this work.
- AVID variants: AVID includes Self-AVID, Cross-AVID, and Joint-AVID, which contrast representations within one modality, across modalities, or using both.
3. Audio-Visual Instance Discrimination
AVID learns visual and audio representations by contrasting cross-modal instances, using memory features and sampled negatives. Its experiments find that Cross-AVID outperforms within-modal variants, while the training setup evaluates multiple objective variants and downstream transfer.
- Goal and Intuition: AVID learns visual and audio representations from corresponding video-audio instances using instance discrimination against other training instances.The representations are optimized to match an instance’s memory features rather than those of other instances.
- Goal and Intuition: Two neural networks independently extract unit-norm video and audio features, while slowly moving memory representations provide contrastive-learning targets.The memory features are maintained for both modalities and used to distinguish corresponding instances from other instances.
- AVID Variants: Cross-AVID contrasts video against audio and audio against video, whereas Self-AVID and Joint-AVID incorporate within-modality discrimination objectives.The three variants optimize different forms of the NCE loss by varying their target representations.
- Training Procedure: NCE trains representations to select their own target from uniformly sampled negative instances using a generalized softmax probability.The negative set contains K other instances drawn uniformly from the training set, and temperature controls distribution softness.
- Experimental Analysis: The analysis evaluates AVID variants through visual and audio transfer tasks, including linear probing on Kinetics and ESC.The setup uses 100K randomly sampled AudioSet videos for pre-training and reports downstream performance for visual and audio features.
- Experimental Analysis: Self-AVID is consistently outperformed by Cross-AVID on both visual and audio tasks, while Joint-AVID also performs worse than Cross-AVID.The authors attribute this pattern to within-modal discrimination being an easier pretext task that can rely partly on low-level statistics.
4. Beyond Instance Discrimination: Cross-Modal Agreement
Cross-Modal Agreement (CMA) extends Cross-AVID beyond isolated instances by grouping samples that agree in both video and audio spaces. These groups provide more reliable positives and negatives, enabling within-modal calibration that addresses feature collapse and improves representations.
- Motivation: Cross-AVID has three limitations: it treats instances independently, can sample false negatives, and does not directly optimize within-modal visual similarity.The authors motivate CMA as a response to these shortcomings.
- Relating instances through agreements: CMA defines an agreement score that is large only when two instances are similar in both video and audio feature spaces.Instances agreeing in both spaces are treated as positives, while instances with poor agreement form negatives.
- CMA Learning Objective: The resulting positive set contains the most jointly similar samples, and the negative set is its complement, allowing self-supervision beyond single instances.During training, negative instances are sampled from the associated negative set.
- CMA Learning Objective: CMA combines cross-modal instance discrimination with within-modal positive discrimination, calibrating similarities between related positive samples rather than comparing each instance with itself.The within-modal term is weighted with the cross-modal term by λ in the combined objective.
- Analyzing CMA: Cross-AVID produces an average random-memory inner product of 0.23 instead of the expected 0, whereas CMA restores the average dot-product to 0.The authors interpret this as CMA addressing the feature-collapsing problem observed for Cross-AVID.
- Analyzing CMA: CMA consistently outperforms within-modal expansion alternatives: audio-only expansion hurts Kinetics performance, while video-only expansion yields only marginal improvements.The authors attribute the advantage to modeling agreement across both feature spaces and obtaining more precise positive sets.
- Analyzing CMA: CMA identifies semantically related videos and rejects visually similar but acoustically different examples, producing more precise positive sets than within-modal expansion methods.For example, audio similarity can distinguish visually similar concepts such as ambulance and bus.
5. Cross-AVID and CMA at scale
The paper evaluates Cross-AVID and CMA on large-scale datasets, including transfer to action recognition and comparisons with prior self-supervised methods. CMA improves linear-probing transfer, while Cross-AVID performs strongly under matched full-finetuning settings.
- Experimental setup: Models are trained on Kinetics-400 and full Audioset, containing 240K and 1.8M video instances, respectively.The visual encoder is an 18-layer R(2+1)D network and the audio encoder is a 9-layer 2D CNN.
- Cross-Modal Agreement: CMA models agreement between video and audio similarities, whereas within-modality expansion alternatives do not model agreement in both spaces.The alternatives expand positive sets using only audio, only video, or either modality; Figure 4 also evaluates transfer and precision.
- Action recognition: CMA yields significant gains across clip-level and video-level Kinetics top-1 accuracy for both metrics and pretraining datasets.Video-level predictions average clip-level predictions from 10 uniformly sampled clips.
- Action recognition: Cross-AVID achieves state-of-the-art performance for equivalent data settings in most UCF-101 and HMDB-51 comparisons.With Audioset pretraining, Cross-AVID exceeds L3 and AVTS by at least 1.0% on UCF and 2.5% on HMDB.
- Action recognition: CMA’s feature-calibration benefits are diminished under full network finetuning, which emphasizes network initialization rather than representation semantic quality.The comparison notes different pretraining and finetuning clip sizes for some methods.
DB ESC DCASE
Audio representations are evaluated by linear classification on ESC-50 and DCASE. Cross-AVID and CMA outperform the cited prior methods on both datasets.
- Evaluation: Audio representations are evaluated on ESC-50 and DCASE using linear probing for sound recognition.ESC-50 results average over five original train/test splits.
6. Discussion
The discussion presents AVID as a cross-modal contrastive method and CMA as an extension that groups related instances using agreement across video and audio spaces. The resulting groups support improved visual and audio representations and strong downstream performance.
- Discussion: AVID contrasts visual representations against multiple audios, and audio representations against multiple videos, rather than relying on within-modal similarity alone.The discussion identifies cross-modal similarity as important for learning from video and audio.
- Discussion: The proposed representations outperform prior work on action recognition and sound classification.The discussion frames these results as evidence for the proposed self-supervised representations.
- Discussion: CMA groups related videos when they agree in both video and audio feature spaces, enabling within-modal similarity optimization alongside cross-modal similarity.It is designed to identify more related videos and reject visually similar videos with different audio.
- Discussion: Cross-modal agreements provide non-trivial correspondences between samples in a multimodal setting.The discussion presents this as a supported implication of CMA’s generalization.
A. Experimental setup
The appendix specifies the architectures, optimization settings, augmentations, and preprocessing used for AVID, CMA, and downstream action-recognition experiments.
- Architecture details: Architecture details for the video and audio networks are provided separately for analysis experiments and comparisons to prior work.These details are organized in Tables 9 and 10.
- Pre-training hyper-parameters: Pretraining optimization and data-augmentation hyperparameters for AVID and CMA are provided in Table 7.The passage identifies Table 7 as the source for these settings.
- Action recognition hyper-parameters: Action-recognition optimization and data-augmentation hyperparameters are provided in Table 8.These settings concern transfer into downstream action-recognition tasks.
- Video pre-processing: Video preprocessing uses 16 fps extraction, random multi-scale cropping, horizontal flipping, and color and temporal jittering.Random cropping uses an 8% minimum area.
- Audio pre-processing: Audio preprocessing loads signals at 24kHz, computes spectrograms with 20ms windows, log-scales and Z-normalizes them, and applies volume and temporal jittering.The spectrogram hop-size is 10ms in Sections 4 and 5 and 20ms in Section 6.
B. Longer AVID pre-training
Cross-AVID was trained for the same number of epochs as AVID+CMA to test whether CMA’s benefits were caused by longer training. Its transfer performance on Kinetics appeared to saturate after 200 pre-training epochs.
- Cross-AVID was trained for the same number of epochs as AVID+CMA to control for longer training as an explanation for CMA’s benefits.
- Cross-AVID transfer performance appeared to have saturated after 200 epochs of pre-training.
C. CMA calibration
The CMA calibration analysis evaluates memory representations from AVID and CMA training on Kinetics with fixed encoders and linear probes. It reports top-1 accuracy for video, audio, and concatenated representations under repeated train/validation splits.
- Memory representations from AVID and CMA trained on Kinetics were compared using linear classifiers while keeping the ConvNet fixed.
- The 220K available training samples were split into 70% training and 30% validation sets, with five sampled splits averaged.
- Linear probes used video, audio, or concatenated video-and-audio representations, with top-1 accuracy measured on validation data.
- CMA models were initialized from the AVID model obtained at epoch 200 during pre-training optimization.
- The analysis experiments used R(2+1)D video networks and Conv2D audio networks with ReLU activations and batch normalization.