Source-linked AI summary
Wav2CLIP: Learning Robust Audio Representations From CLIP
Ho-Hsiang Wu, Prem Seetharaman, Kundan Kumar, Juan Pablo Bello
TL;DR
The paper addresses how to learn robust, transferable audio representations without relying on extensive labeled audio or jointly training visual and auditory models. Wav2CLIP distills frozen CLIP image embeddings into an audio encoder and evaluates the resulting shared-space representation across downstream and multimodal tasks. It performs competitively with publicly available audio representations, reaches similar supervised performance with roughly one-tenth the labeled data in a VGGSound setting, and supports zero-shot and cross-modal applications.
Problem
Robust audio representations should transfer across diverse tasks, but supervised learning requires expensive labeled data and prior audio-visual methods jointly learn visual and auditory models.
Method
Wav2CLIP freezes CLIP’s vision model and trains an audio encoder to predict its image embeddings from video audio using contrastive distillation.
Results
Wav2CLIP performs slightly better or similarly to YamNet and OpenL3 in almost all classification tasks, while reaching competitive retrieval and captioning performance.
Takeaways & Limitations
The shared embedding space supports zero-shot audio classification, cross-modal retrieval, audio captioning, and image generation from audio.
Takeaways & Limitations
Wav2CLIP struggles to disambiguate singing from speech when both share similar visual characteristics, and struggles where co-occurring environments create ambiguous pre-training examples.
Abstract
from arXiv · showhide
We propose Wav2CLIP, a robust audio representation learning method by distilling from Contrastive Language-Image Pre-training (CLIP). We systematically evaluate Wav2CLIP on a variety of audio tasks including classification, retrieval, and generation, and show that Wav2CLIP can outperform several publicly available pre-trained audio representation algorithms. Wav2CLIP projects audio into a shared embedding space with images and text, which enables multimodal applications such as zero-shot classification, and cross-modal retrieval. Furthermore, Wav2CLIP needs just ~10% of the data to achieve competitive performance on downstream tasks compared with fully supervised models, and is more efficient to pre-train than competing methods as it does not require learning a visual model in concert with an auditory model. Finally, we demonstrate image generation from Wav2CLIP as qualitative assessment of the shared embedding space. Our code and model weights are open sourced and made available for further applications.
1. INTRODUCTION
Wav2CLIP distills CLIP into an audio representation aligned with image and text embeddings. This enables broad audio, multimodal, and zero-shot applications while simplifying training.
- Motivation: Self-supervised learning leverages intrinsic data structure and large unlabeled datasets, reducing dependence on expensive human annotation for transferable representations.Pre-trained models can be fine-tuned for downstream tasks with less data.
- Contribution: Wav2CLIP distills CLIP into an audio-visual correspondence model by predicting CLIP embeddings from audio alone.The CLIP vision model is frozen during training, making the recipe lightweight and scalable to large video datasets.
- Contribution: Its CLIP-derived embeddings align audio with images and text, enabling zero-shot audio classification, audio captioning, and cross-modal retrieval.The shared space supports applications such as searching for sounds in surveillance videos or Foley effects in film editing.
- Evaluation: Wav2CLIP is systematically evaluated across classification and retrieval tasks, with multimodal transfer and image generation used to assess the embedding space.The paper uses VQGAN-CLIP to guide image generation directly from audio.
2. METHOD
The method uses a two-stage distillation process: CLIP image embeddings provide the audio pretext target, and a contrastive objective trains audio representations in the shared multimodal space.
- CLIP foundation: CLIP learns shared image-text embeddings from paired internet data using noise-contrastive estimation.Same-sample image-text pairs are positives, while other examples in the batch serve as negatives.
- Wav2CLIP distillation: Wav2CLIP pre-trains an audio encoder by distilling CLIP image embeddings obtained from videos.The approach uses a two-stage pipeline and experiments with MLP projection layers.
- Wav2CLIP distillation: Cross-projection loss combines L(f(Image), Audio) and L(Image, g(Audio)) to stabilize distillation and enforce multimodal consistency.The projection functions provide additional trainable MLP flexibility while allowing audio embeddings to recover CLIP image embeddings.
3. EXPERIMENTAL DESIGN
The experiments freeze the learned audio encoder and evaluate it as a feature extractor across diverse classification, retrieval, captioning, and multimodal tasks using varied datasets and baselines.
- Model setup: The audio encoder converts waveforms to spectrograms, uses ResNet-18, and outputs 512-dimensional embeddings through average pooling.Frame-level tasks use 1-second non-overlapping audio segments.
- Evaluation protocol: Downstream evaluation freezes the learned weights and trains supervised classifiers from scratch as reference baselines.The study selects datasets varying in clip count, category count, and task type.
- Datasets and tasks: The benchmark covers multi-class, multi-label, zero-shot, audio retrieval, cross-modal retrieval, and audio captioning across datasets including ESC-50, VGGSound, TAU, DESED, and Clotho.The datasets span simple, large-scale, multilabel, acoustic-scene, segment-retrieval, and captioning settings.
- Multimodal evaluation: Cross-modal retrieval and zero-shot classification compare cosine distances between Wav2CLIP audio embeddings and CLIP image or text embeddings.Audio captioning freezes the encoder and trains a 1-layer transformer decoder, with COCO caption metrics reported.
- Baselines: Comparisons include supervised training from scratch, OpenL3, YamNet, and task-specific state-of-the-art systems.The study also evaluates VGGSound classifiers using 1, 2, 5, 10, 20, 50, and 100% of the training split.
4. RESULTS AND DISCUSSIONS
Across classification, retrieval, and captioning, Wav2CLIP provides competitive frozen representations and often matches or approaches supervised and pretrained baselines, while revealing task-specific strengths and limitations.
- Classification: Wav2CLIP performs slightly better or similarly to YamNet and OpenL3 in almost all classification tasks, except FSD50K.On TAU, OpenL3 and Wav2CLIP outperform YamNet, consistent with their audio-visual self-supervision.
- Limitations: Wav2CLIP remains below task-specific SOTA models because evaluation uses a frozen encoder with only simple MLP classifiers, despite requiring one encoder version across tasks.Zero-shot classification is non-random but weaker than MLP classifiers, especially for many-class and multi-label settings such as FSD50K.
- Retrieval: Wav2CLIP reaches ∼0.05 MRR for cross-modal retrieval, retrieving relevant clips from the top 20 on average among 15k clips.OpenL3 performs worse despite audiovisual pre-training because it is not forced into the shared embedding space.
- Audio Captioning: Wav2CLIP slightly outperforms the audio-captioning baseline on all reported metrics, although the comparison uses different encoder and decoder architectures.Table 3 reports Bleu1/4, METEOR, ROUGEL, CIDEr, SPICE, and SPIDEr.
- Classification: ∼1/10 amount of data lets Wav2CLIP reach similar VGGSound classification performance as the supervised counterpart in the smaller labeled-data region.The comparison includes Supervise, OpenL3, YamNet, and Wav2CLIP across different training-data percentages.
- YamNet vs Wav2CLIP: Wav2CLIP’s confusion patterns show strengths for visually distinct environments but weaknesses when visual characteristics overlap or co-occur during pre-training.It better separates Airport, Shopping mall, and several transport-related environments, but struggles with Park versus Street traffic and singing versus speech.
5. IMAGE GENERATION: LISTENING AT A GLANCE
Wav2CLIP audio embeddings guide VQGAN-CLIP to generate images, producing semantically plausible results and showing compositional behavior for mixtures of songs.
- VQGAN-CLIP uses Wav2CLIP audio embeddings instead of text or image prompts to guide image generation.CLIP embeddings guide searches through VQGAN's latent space; Wav2CLIP audio embeddings replace the original prompts.
- Plausible UrbanSound8K images with reasonable semantic meanings indicate that Wav2CLIP audio embeddings occupy a meaningful shared space.
- Images generated from individual musdb18 tracks and mixtures show compositionality, with mixture images containing elements from individual components.
- For similar instruments, diverse images arise from noise introduced in the latent space during fine-tuning on each prompt.
6. CONCLUSION
Wav2CLIP distills audio representations from CLIP and is evaluated across diverse downstream tasks. The resulting embeddings are reported as general, robust, and effective across audio classification, retrieval, and multimodal applications.
- Wav2CLIP distills audio representations from CLIP and evaluates the learned embeddings across diverse downstream tasks.
- Wav2CLIP outputs general and robust audio representations and performs well across audio classification and retrieval tasks compared with YamNet and OpenL3.
- Wav2CLIP embeddings can be transferred to multimodal tasks, including cross-modal audio-to-image generation.