PaperlayerView exact revision on arXiv

Source-linked AI summary

Contrastive Audio-Visual Masked Autoencoder

Yuan Gong, Andrew Rouditchenko, Alexander H. Liu, David Harwath, Leonid Karlinsky, Hilde Kuehne, James Glass

arXiv:2210.07839v4cs.MMcs.CVcs.SDeess.AS

TL;DR

Self-supervised audio-visual learning needs to use large unlabeled video collections because manual annotation is expensive and difficult to scale. The paper extends MAE to audio-visual inputs and proposes CAV-MAE, combining masked data modeling with contrastive learning to learn joint and coordinated representations. CAV-MAE reaches 65.9% accuracy on VGGSound and is comparable with the previous best supervised pretrained model on AudioSet for audio-visual event classification.

  • Problem

    Manual audio-video annotation is expensive and difficult to scale, motivating self-supervised learning from web-scale unlabeled video while addressing audio-visual fusion and retrieval.

  • Method

    CAV-MAE combines contrastive audio-visual learning with masked data modeling in a single model to learn joint and coordinated audio-visual representations.

  • Results

    65.9% accuracy on VGGSound is a new SOTA, and CAV-MAE is comparable with the previous best supervised pretrained model on AudioSet for audio-visual event classification.

  • Takeaways & Limitations

    The contrastive objective supports audio-visual retrieval while also helping CAV-MAE learn a stronger joint representation for event classification.

  • Takeaways & Limitations

    Vanilla AV-MAE lacks an explicit audio-visual correspondence objective and may process distinct modalities sub-optimally with shared joint-encoder weights.

Abstract

from arXiv · show

In this paper, we first extend the recent Masked Auto-Encoder (MAE) model from a single modality to audio-visual multi-modalities. Subsequently, we propose the Contrastive Audio-Visual Masked Auto-Encoder (CAV-MAE) by combining contrastive learning and masked data modeling, two major self-supervised learning frameworks, to learn a joint and coordinated audio-visual representation. Our experiments show that the contrastive audio-visual correspondence learning objective not only enables the model to perform audio-visual retrieval tasks, but also helps the model learn a better joint representation. As a result, our fully self-supervised pretrained CAV-MAE achieves a new SOTA accuracy of 65.9% on VGGSound, and is comparable with the previous best supervised pretrained model on AudioSet in the audio-visual event classification task. Code and pretrained models are at https://github.com/yuangongnd/cav-mae.

1 INTRODUCTION

The paper addresses self-supervised audio-visual learning from unlabeled videos by extending masked autoencoding and combining it with contrastive correspondence learning. CAV-MAE learns joint and coordinated representations, improving classification and retrieval performance.

  • Self-supervised audio-visual learning seeks to exploit web-scale unlabeled video because manually annotating audio and video is expensive and difficult to scale.
  • Contrastive audio-visual learning brings paired audio and visual representations closer than mismatched ones, supporting coordinated representations for cross-modal retrieval.
  • Masked data modeling learns representations by reconstructing original inputs or features from corrupted ones.
  • AV-MAE extends single-modal MAE to fuse audio and visual signals through cross-modal masked data modeling, allowing one modality to be reconstructed using the other.
  • Contrastive learning uses audiovisual pair information but may discard modality-unique information, whereas AV-MAE preserves more input information without an explicit correspondence objective.
  • CAV-MAE combines contrastive learning with masked data modeling to learn joint and coordinated representations in one model.
  • 65.9% accuracy on VGGSound is reported as a new SOTA, while CAV-MAE is comparable with the previous best supervised pretrained model on AudioSet.

2 CONSTRASTIVE AUDIO-VISUAL MASKED AUTOENCODER

CAV-MAE combines contrastive audio-visual learning with cross-modal masked data modeling, extending masked autoencoders to audio-visual inputs while preserving separate modality streams and a joint reconstruction stream.

  • Contrastive audio-visual learning: The conventional contrastive model independently encodes audio and visual tokens, producing mean-pooled modality representations for contrastive learning.Audio and visual tokens are processed by separate Transformer encoders before contrastive loss is applied.
  • Vanilla audio-visual masked autoencoder: AV-MAE extends MAE from single-modality inputs to audio-visual pairs by masking 75% of tokens and reconstructing masked audio and visual inputs.The joint encoder receives unmasked tokens, while a joint decoder reconstructs normalized modality inputs using mean square error.
  • Design motivation: AV-MAE can model cross-modal reconstruction, but lacks an explicit paired-correspondence objective and processes both modalities with the same joint-encoder weights.The paper identifies these choices as potential limitations of the vanilla design.
  • CAV-MAE: CAV-MAE combines contrastive audio-visual learning with cross-modal masked data modeling to learn joint and coordinated audio-visual representations.Its loss sums reconstruction loss with a weighted contrastive loss.
  • CAV-MAE architecture: CAV-MAE uses independent audio and visual streams alongside a joint audio-visual stream, sharing attention and MLP weights while using stream-specific normalization layers.The single-modality stream outputs support contrastive learning, and the multimodal stream output supports reconstruction.
  • Masked contrastive learning: Masked contrastive audio-visual learning randomly masks inputs before contrastive learning; at a 75% masking ratio, matching accuracy is about 72%.The effective contrastive batch size is 27, or 108 across 4 GPUs, and the authors describe the task as neither trivial nor impossible.

3 SELF-SUPERVISED MODEL PRETRAINING

The paper pretrains and compares several audio-visual and single-modal self-supervised models under a shared pipeline, with CAV-MAE combining contrastive and reconstruction objectives.

  • The study compares Audio-MAE, Visual-MAE, CAV, Vanilla AV-MAE, AV-MAE, CAV-MAE, and a scaled CAV-MAE variant.The variants isolate single-modal pretraining, contrastive learning, masked data modeling, modal-specific encoders, and scaling effects.
  • Except for CAV-MAEscale+, models use the same batch size, pretraining duration, and full AudioSet-2M data.The standard pipeline uses batch size 48 for 12 epochs, without class-balanced sampling.
  • The pretraining process is fully self-supervised, including the ImageNet-pretrained weight initialization.

4 AUDIO-VISUAL EVENT CLASSIFICATION

The paper evaluates pretrained representations for audio-visual and single-modal event classification across datasets and data regimes. CAV-MAE combines complementary objectives and achieves strong performance under fully self-supervised, comparatively low-resource settings.

  • Evaluation setup: The evaluation fine-tunes representations on AudioSet-20K, AudioSet-2M, and VGGSound, covering different data volumes and domain conditions.
  • Evaluation setup: Fine-tuning retains the pretrained encoder and adds a randomly initialized linear classification head with separate learning-rate scales.The new head uses a 10×-100× larger learning rate than pretrained weights.
  • Objective ablation: 2.0 and 3.1 mAP are the respective boosts from CAV-MAE over CAV and AV-MAE on AudioSet-20K.The models share the same fine-tuning architecture; their difference is the self-supervised objective.
  • Single-modal transfer: 37.7 mAP audio and 19.8 mAP visual performance show gains over 34.2 and 15.7 mAP from corresponding single-modal models on AudioSet-20K.The comparison uses single-modal fine-tuning after multimodal CAV-MAE pretraining.
  • Benchmark results: CAV-MAE performs better than MBT on VGGSound and comparably on AudioSet-20K and AudioSet-2M for audio-visual classification.For audio-based classification, it performs better than Audio-MAE on AudioSet-20K and comparably on AudioSet-2M.
  • Ablations: Modal-specific encoders improve performance, larger batches and longer pretraining generally help, and the performance margin shrinks on larger fine-tuning datasets.The contrastive-loss weight also has a substantial effect, with overly large or small values reducing performance.

5 AUDIO-VISUAL RETRIEVAL

The retrieval evaluation tests whether CAV-MAE captures audio-visual correspondences across AudioSet and VGGSound. Contrastive learning is necessary for retrieval, while masked modeling and scaling can further improve it.

  • Evaluation: Retrieval is evaluated on subsets of 1,725 AudioSet and 1,545 VGGSound audio-visual samples.The subsets represent about 10% of each evaluation set and support construction of manageable similarity matrices.
  • Evaluation: Table 3 reports quantitative visual-to-audio retrieval results, while additional audio-to-visual results and experiments are provided elsewhere.
  • Findings: Contrastive learning is necessary for audio-visual retrieval because Vanilla-MAE and AV-MAE perform close to random guessing.
  • Findings: Cross-modal masked data modeling does not hurt retrieval and often improves it, with CAV-MAE generally outperforming CAV when λc = 0.1.
  • Findings: Scaling the pretraining batch size and number of epochs improves retrieval performance.

6 RELATED WORK

Related work centers on self-supervised learning from natural audio-visual pairings and masked data modeling. Prior methods include correspondence-based approaches and modality-specific adaptations of masked auto-encoders.

  • Contrastive Audio-Visual Learning: Audio-visual self-supervision exploits natural pairings through knowledge distillation, paired sample discrimination, and contrastive learning.
  • Contrastive Audio-Visual Learning: Contrastive audio-visual learning produces coordinated representations that bring paired samples closer than mismatched samples, supporting cross-modal retrieval.
  • Masked Auto-Encoder: Masked data modeling learns representations by reconstructing original inputs or features from corrupted versions.
  • Masked Auto-Encoder: Masked auto-encoders have been applied across visual and audio domains, with recent work adapting the MAE pipeline to audio.

7 CONCLUSION

The paper introduces CAV-MAE, which combines masked data modeling and contrastive learning to learn joint, coordinated audio-visual representations. It reports strong performance across joint classification, retrieval, and single-modal audio classification.

  • CAV-MAE combines masked data modeling and contrastive learning as complementary objectives for audio-visual self-supervised learning.The design aims to learn a joint and coordinated audiovisual representation with one model.
  • CAV-MAE uses a multi-stream forward pass, joint-specific encoder architecture, and masked contrastive learning to combine the two frameworks while avoiding representation collapse.
  • CAV-MAE supports both audio-visual joint event classification and audio-visual retrieval tasks.
  • CAV-MAE matches or outperforms state-of-the-art models on audio-visual event classification with fully self-supervised pretraining and fewer computational resources.
  • CAV-MAE is comparable to contrastive-only models on retrieval and achieves new state-of-the-art performance on audio-based event classification through multimodal pretraining.

ETHICS STATEMENT

The paper uses publicly available YouTube videos and reports implementation settings intended to support reproducibility. It also acknowledges that the model could be used in security applications, including malicious surveillance.

  • The experiments use publicly available YouTube videos and exclude videos removed by users.
  • The paper notes that the model can support security-related applications but could also be used for malicious surveillance, motivating careful distribution of code and models.
  • The experiments use AudioSet and VGGSound, with released training and test sample IDs intended to improve reproducibility despite changing video availability.
  • Most experiments run on 4 NVIDIA GTX Titan X Pascal GPUs, while CAV-MAEScale+ uses 4 NVIDIA RTX A5000 GPUs.
  • CAV-MAE pretraining takes about one week with 4 GPUs.
  • The full encoder-decoder has approximately 190M parameters, while downstream encoders use approximately 160M or 85M parameters depending on modality.

C AUDIO-VISUAL ACTION RECOGNITION EXPERIMENTS

The action-recognition experiments evaluate CAV-MAE on Kinetics-Sounds under in-domain and MBT comparison settings. CAV-MAE consistently outperforms its CAV and AV-MAE counterparts and matches or outperforms MBT under fully self-supervised learning.

  • Kinetics-Sounds is selected because existing audio-visual action-recognition datasets are often visual-heavy and dominated by the visual branch.
  • The first experiment pretrains and fine-tunes CAV, AV-MAE, and CAV-MAE on Kinetics-Sounds and reports Top-1 validation accuracy without AudioSet pretraining.
  • The Kinetics-Sounds experiments use a 32-class label set rather than the dataset’s original 34-class version.
  • CAV-MAE performs better than both CAV and AV-MAE on Kinetics-Sounds, consistent with the AudioSet and VGGSound experiments.
  • The MBT comparison reports top-1 accuracy using both the unrestricted Kinetics-400 label set and the restricted 32-class Kinetics-Sounds label set.

D.3 MSR-VTT DATASET RETRIEVAL EXPERIMENTS

The MSR-VTT experiments evaluate supervised-training and zero-shot audio-visual retrieval, while additional analyses examine initialization and masking. CAV-MAE matches or improves on contrastive-only models, but retrieval varies with data scale and masking ratio.

  • The MSR-VTT experiments compare audio-visual retrieval in both directions using models trained on MSR-VTT and models evaluated zero-shot after AudioSet pretraining.
  • CAV and CAV-MAE outperform existing methods in both retrieval directions when trained with MSR-VTT data.
  • The MAE objective does not hurt, and can improve, retrieval performance relative to CAV when λc is held constant.
  • In zero-shot MSR-VTT retrieval, CAV-MAE achieves similar visual-audio retrieval results to methods trained on 100M HowTo100M examples despite using 2M AudioSet examples.
  • ImageNet initialization improves performance, but without it CAV-MAE is only 1.0% mAP lower on AudioSet-2M and initialization is not indispensable.
  • ImageNet supervised weights improve AudioSet-20K fine-tuning by 3.7% over self-supervised weights, making direct comparisons with supervised-pretrained models less fair.
  • Audio-visual joint classification is not sensitive to masking ratio, whereas retrieval performance decreases at higher ratios, especially when masking is very high.

F.2 IMPACT OF AUDIO TRAINING MASKING STRATEGY

Audio masking strategy has little effect on joint event classification but substantially affects retrieval and reconstruction. Structured masking makes reconstruction harder, while time-frequency training can improve retrieval and reconstruction under some settings.

  • Masking strategies: Uniform, unstructured masking is the default strategy, while time, frequency, and time-frequency masking are evaluated as alternatives for audio inputs.The paper uses uniform masking throughout except in the masking-strategy study.
  • Joint event classification: All four training masking strategies produce similar joint event classification performance at a 75% training masking ratio.Time-frequency masking at a 50% ratio yields slightly better performance on AudioSet-20K and VGGSound.
  • Audio-visual retrieval: Frequency and time-frequency masking achieve the best retrieval performance, whereas uniform masking produces the worst performance at the same 75% ratio.Retrieval performance also decreases as the masking ratio increases under uniform unstructured masking.
  • Frame sampling: Classification performance consistently improves as more uniformly sampled video frames are used, but the gain saturates across Kinetics-Sounds, AudioSet-20K, and VGGSound.The paper’s standard setting samples 10 frames from each 10-second clip at 1 FPS.
  • Spectrogram reconstruction: CAV-MAE retains strong reconstruction ability at a 90% inference masking ratio, including on VGGSound samples from a dataset different from pretraining.The model used AudioSet pretraining with a 75% masking ratio without target normalization.
  • Spectrogram reconstruction: Structured masking generally increases reconstruction MSE, with time masking hardest and uniform unstructured masking easiest.The pattern indicates that the model relies on local neighboring unmasked information when reconstructing masked spectrogram regions.
  • Spectrogram reconstruction: A model trained with time-frequency masking reconstructs audio spectrograms better than one trained with uniform masking, especially under time and frequency inference masking.Training strategy has a larger effect on reconstruction than on downstream classification.
  • Visual sound source localization: Visual sound source localization is not strong despite good retrieval, and some examples localize sound to background regions rather than the main sound source.The authors hypothesize that masked contrastive learning can encourage reliance on contextual information when the main source is masked.

J IMPACT OF THE AUDIO-VISUAL PAIRING INFORMATION IN TRAINING DATASET

The study examines how audio-visual pairing in the training data affects CAV-MAE. Its comparisons distinguish the roles of contrastive correspondence learning and reconstruction, while retrieval examples provide qualitative evaluation.

  • Pairing information: AV-MAE can reconstruct one modality using information from the other, but its use of audio-visual pairing information without an explicit correspondence objective is unknown.The reconstruction pathway theoretically permits audio-visual correlation learning, but does not directly encourage paired correspondence.
  • Pairing information: Using only the contrastive objective produces worse performance and lower robustness to noise in the training set.This comparison isolates contrastive learning without the reconstruction objective.
  • Pairing information: Using only the reconstruction objective does not effectively exploit audio-visual pair information.The result contrasts with the explicit correspondence signal supplied by contrastive learning.
  • Pairing information: Table 16 compares joint event classification for models trained with original audio-visual pairs and randomly shuffled audio-visual pairs.The supplied passage identifies the comparison but does not report its numerical outcome.
  • Cross-modal retrieval: Zero-shot audio-to-image and image-to-audio retrieval results are presented on VGGSound using one modality as the query or key.Paired images are shown for audio queries because spectrograms are difficult to read, while audios are shown for image-to-audio retrieval.
  • Reconstruction examples: CAV-MAE reconstruction samples are shown at 50%, 75%, and 90% masking, with samples drawn from VGGSound although pretraining uses AudioSet.The reconstruction model is pretrained with a 75% masking ratio without target normalization.
  • Reconstruction examples: Audio spectrogram reconstruction is compared across uniform and time-frequency training masks under multiple inference masking settings using reconstruction MSE.The comparison is shown for both 75% and 90% inference masking ratios.
Loading 2210.07839v4…