PaperlayerView exact revision on arXiv

Source-linked AI summary

AudioCLIP: Extending CLIP to Image, Text and Audio

Andrey Guzhov, Federico Raue, Jörn Hees, Andreas Dengel

arXiv:2106.13043v1cs.SDcs.CVeess.AS

TL;DR

AudioCLIP addresses limited labeled audio data and the rarity of models combining more than two modalities by extending CLIP with audio. It integrates ESResNeXt through AudioSet, enabling tri-modal classification and querying, and reports state-of-the-art environmental sound classification plus zero-shot results. The paper also evaluates cross-modal querying, training variants, and broader future evaluation needs.

  • Problem

    Limited qualitatively labeled data constrains audio research, while combining more than two modalities remains uncommon in the audio domain.

  • Method

    AudioCLIP integrates the ESResNeXt audio model into CLIP and trains text, image, and audio heads together using AudioSet.

  • Results

    AudioCLIP achieves state-of-the-art classification accuracies of 90.07% on UrbanSound8K and 97.15% on ESC-50, with zero-shot results of 68.78% and 69.40%, respectively.

  • Takeaways & Limitations

    The model supports modality-specific classification and cross-modal querying across text, images, and audio while generalizing to unseen datasets in a zero-shot setup.

  • Takeaways & Limitations

    The authors identify evaluation on a wider variety of datasets and tasks, and more powerful image and audio backbones, as future directions.

Abstract

from arXiv · show

In the past, the rapidly evolving field of sound classification greatly benefited from the application of methods from other domains. Today, we observe the trend to fuse domain-specific tasks and approaches together, which provides the community with new outstanding models. In this work, we present an extension of the CLIP model that handles audio in addition to text and images. Our proposed model incorporates the ESResNeXt audio-model into the CLIP framework using the AudioSet dataset. Such a combination enables the proposed model to perform bimodal and unimodal classification and querying, while keeping CLIP's ability to generalize to unseen datasets in a zero-shot inference fashion. AudioCLIP achieves new state-of-the-art results in the Environmental Sound Classification (ESC) task, out-performing other approaches by reaching accuracies of 90.07% on the UrbanSound8K and 97.15% on the ESC-50 datasets. Further it sets new baselines in the zero-shot ESC-task on the same datasets (68.78% and 69.40%, respectively). Finally, we also assess the cross-modal querying performance of the proposed model as well as the influence of full and partial training on the results. For the sake of reproducibility, our code is published.

1 Introduction

AudioCLIP addresses limited qualitative audio labels and the rarity of audio models combining more than two modalities by adding audio to CLIP. The resulting tri-modal model targets classification, zero-shot inference, and cross-modal querying.

  • Motivation: Limited qualitatively labeled data constrains both unimodal and multimodal audio research and motivates zero-shot and few-shot contrastive approaches.These approaches rely on textual descriptions to address data scarcity.
  • Contribution: AudioCLIP combines the high-performance ESResNeXt audio model with CLIP to create a tri-modal hybrid architecture.The added audio modality extends CLIP beyond its original text-image design.
  • Capabilities: Joint use of text, images, and audio supports environmental sound classification, audio zero-shot inference, and cross-modal querying in any modality combination.The model extends zero-shot capabilities from CLIP to audio while supporting cross-modal querying.
  • Scope: The paper evaluates the proposed model and its training variants, then reports results and follow-up research directions.The evaluation includes the influence of partial and full training on classification and querying.

2 Related Work

Related work moves from standalone environmental sound classification toward multimodal and contrastive methods. AudioCLIP builds on this trajectory by training across text, visual, and audible modalities for classification, querying, and zero-shot generalization.

  • Environmental Sound Classification: Environmental sound classification assigns everyday sound recordings to labels such as alarms, horns, jackhammers, and animal sounds.Prior systems used one- or two-dimensional CNNs over static or trainable time-frequency representations.
  • Multimodal Approaches: Multimodal audio approaches initially combined audio with one additional modality, while recent studies suggest benefits from using more modalities.Earlier audio-related multimodal systems utilized at most two modalities simultaneously.
  • Contrastive Learning: Contrastive learning helps address the lack of qualitatively labeled data and supports zero-shot classification tasks.The related approaches use contrastive learning as a form of self-supervised learning with textual descriptions.
  • AudioCLIP: AudioCLIP applies contrastive learning to textual, visual, and audible modalities for modality-specific classification, querying, and zero-shot generalization.The model is presented as a broader multimodal querying system rather than only an audio classifier.

3 Model

AudioCLIP combines CLIP’s text-image encoders with an ESResNeXt audio encoder in a shared tri-modal architecture. It aligns all modality pairs through added audio-related losses and supports simultaneous or pairwise processing.

  • Architecture: The hybrid architecture combines a ResNet-based CLIP model for visual and textual modalities with ESResNeXt for audio.The model is illustrated as an extension of the original text-image CLIP workflow.
  • CLIP Backbone: CLIP jointly trains text and image heads to align matching representations in a shared multimodal embedding space.Matching text-image pairs are encouraged to have higher cosine similarity than incorrect pairs using symmetric cross-entropy loss.
  • CLIP Backbone: The CLIP text encoder uses a modified Transformer, while the selected image encoder is a modified ResNet-50 chosen for lower computational complexity.The text input uses byte-pair encoding with a vocabulary of 49 408 and maximum sequence length 76.
  • Audio Encoder: ESResNeXt is a ResNeXt-50-based audio model with trainable time-frequency transformations and approximately 30 M parameters.It was selected for competitive AudioSet performance and state-of-the-art-level results on UrbanSound8K and ESC-50.
  • Hybrid Model – AudioCLIP: AudioCLIP adds audio as an equally important modality and trains text-to-audio and image-to-audio similarities alongside text-to-image similarity.The model processes all three modalities simultaneously or any pair of them.

4 Experimental Setup

The experiments use multiple image, audio, and mixed datasets to initialize, train, and evaluate AudioCLIP. AudioSet connects video frames, audio tracks, and class names for tri-modal training, while UrbanSound8K and ESC-50 support zero-shot and fine-tuned evaluation.

  • Datasets: Five image, audio, and mixed datasets are used directly or indirectly for training and evaluation.Their roles include weight initialization, hybrid training, and target evaluation tasks.
  • Datasets: The Composite CLIP Dataset initializes the text and image heads, while ImageNet initializes ESResNeXt and serves as a zero-shot target.The Composite CLIP Dataset contains roughly 400 M text-image pairs, and ImageNet contains more than 1 M images across 1 000 classes.
  • Datasets: AudioSet provides approximately 1.8 M training examples and a roughly 20 k evaluation set across 527 non-exclusive audio classes.The samples are video snippets up to 10 seconds long.
  • AudioSet Training: AudioSet becomes the link between CLIP and AudioCLIP by supplying video frames, audio tracks, and class names for hybrid-model input.Audio tracks and labels first support image-to-audio transfer learning for ESResNeXt.
  • Evaluation: Training samples one randomly selected frame from ten equally distant video frames, whereas evaluation uses only the central frame.Performance metrics are reported on the AudioSet evaluation set.
  • Evaluation: UrbanSound8K and ESC-50 are evaluated with AudioSet-trained zero-shot inference and audio-head fine-tuning in standalone and cooperative settings.UrbanSound8K has 8 732 tracks in ten classes, while ESC-50 has 2 000 five-second tracks in 50 classes and five folds.

4.2 Data Augmentation

Because the audio datasets contain far fewer samples than the composite CLIP dataset, the training pipeline applies several augmentations to reduce overfitting and align inputs.

  • Two orders of magnitude fewer training samples in the audio datasets make overfitting a concern, especially for UrbanSound8K and ESC-50.
  • Time scaling randomly changes track duration and pitch using factors from approximately U[-1.5, 1.5].
  • Time inversion randomly flips tracks along the time axis with probability 0.5.
  • Random cropping or padding aligns samples to the longest non-augmented track, while evaluation uses center cropping or padding.
  • Additive white Gaussian noise is sampled with a random signal-to-noise ratio from 10.0 dB to 120 dB.

4.3 Training

AudioCLIP is assembled by combining pretrained text-image CLIP and ESResNeXt audio representations, followed by staged and joint training across modalities.

  • A ResNet-based CLIP text-image model is combined with an ImageNet-initialized, AudioSet-pretrained ESResNeXt audio model.
  • The audio head is first pretrained on AudioSet, then the model continues training jointly in a tri-modal setting.
  • AudioCLIP training uses audio snippets, corresponding video frames, and assigned textual labels from AudioSet.
  • The audio head is fine-tuned on UrbanSound8K and ESC-50 using audio and corresponding text labels, while evaluation covers ImageNet and three audio datasets.
  • During cooperative pretraining, the audio classification layer is replaced with a randomly initialized layer matching CLIP’s embedding-space size.
  • Frozen text and image heads act as teachers in a multimodal knowledge-distillation setup while the audio head learns compatible outputs.

AudioCLIP Training

AudioCLIP addresses distribution mismatch and domain-specific classification through whole-model AudioSet training and downstream audio-head fine-tuning.

  • Joint audio-head training makes its outputs compatible with vanilla CLIP, but AudioSet’s image-text distributions may differ from CLIP’s.
  • Whole tri-modal training tunes all modality-specific heads on AudioSet video frames, class names, and audio samples.
  • AudioCLIP supports general multimodal querying, while domain-specific tuning targets concepts that differ only slightly.
  • The audio encoding head is tuned to UrbanSound8K and ESC-50 for downstream evaluation.
  • The AudioSet-pretrained ESResNeXt audio head is separately fine-tuned on both downstream datasets to assess classification-accuracy changes.
  • Downstream AudioCLIP fine-tuning updates only the audio head, freezes text and image heads, and uses one textual class label per sample.

4.4 Hyper-Parameters

The training configuration uses SGD with Nesterov momentum, scheduled learning rates, and different epoch counts for AudioSet training and downstream fine-tuning.

  • Training uses SGD with Nesterov momentum 0.9, weight decay 5 · 10^-4, and batch size 64.
  • Table 1 evaluates ESResNeXt trained on AudioSet for more epochs and reports improved performance compared with the original training.
  • The learning rate changes from 10^-4 with decay factor 0.95 during standalone pretraining to 5 · 10^-5 with factor 0.98 during downstream fine-tuning.
  • AudioSet-based training runs for 30 epochs, while downstream fine-tuning runs for 50 epochs.

4.5 Performance Evaluation

AudioCLIP was evaluated on classification and cross-modal querying, with classification tested using either the audio-head or the full multimodal model. Querying was assessed only for the multimodal network across multiple datasets and modalities.

  • Classification: Classification was evaluated with both AudioCLIP and its ESResNeXt audio-head.The audio-head predicts dataset class labels directly, whereas AudioCLIP constructs targets from textual labels.
  • Classification: AudioCLIP classification included an intermediate target-construction step from textual labels.
  • Classification: The evaluation covered trained and zero-shot classification on UrbanSound8K and ESC-50 after AudioSet pre-training.
  • Querying: Querying used image and/or audio queries with text as the result representation, making classification a querying sub-task.
  • Querying: Querying performance was assessed on ImageNet, AudioSet, UrbanSound8K, and ESC-50 using P@1/R@1 and mAP.
  • Training comparison: Table 2 compares AudioCLIP after partial audio-head training and full training on AudioSet.

5 Results

AudioCLIP improved environmental sound classification through extended, partial, and full training, while full training generally improved querying on AudioSet and downstream tasks but could reduce performance on ImageNet.

  • Audio-head training: 34.14% mAP versus 28.17% followed extended AudioSet pre-training, with downstream audio-head accuracies of 89.49% on UrbanSound8K and 95.90% on ESC-50.The extended schedule used 30 epochs instead of 5.
  • AudioCLIP training: Tri-modal training reduced audio-head overfitting by diversifying its target distribution through video frames.
  • Partial training: Partial training achieved 89.95% accuracy on UrbanSound8K and 96.65% on ESC-50, exceeding current state-of-the-art results reported in the passage.
  • Full training: Full training improved over partial training and reached 90.07% accuracy on UrbanSound8K and 97.15% on ESC-50.
  • Zero-shot classification: Full training set a 68.78% zero-shot classification baseline on UrbanSound8K.
  • Image querying: Full training increased AudioSet querying mAP but decreased ImageNet querying performance, likely because the dataset distributions differ.
  • Audio querying: Full training increased querying performance on AudioSet and UrbanSound8K, but not on ESC-50, where the gap was close to marginal.
  • Cross-modal querying: Full training benefited both audio-by-image and image-by-audio querying in terms of mAP.

6 Conclusion

AudioCLIP extends CLIP with audio to support tri-modal classification and querying. It achieves state-of-the-art environmental sound classification and improved zero-shot results, while broader evaluation remains future work.

  • Contribution: AudioCLIP extends CLIP from text and images to audio using the ESResNeXt sound-classification model.
  • Classification results: 90.07% accuracy on UrbanSound8K and 97.15% on ESC-50 established new state-of-the-art classification results.
  • Zero-shot results: Zero-shot inference reached 69.40% on ESC-50 and established a 68.78% baseline on UrbanSound8K.
  • Evaluation: The authors evaluated cross-modal querying and the effects of partial and full training on classification and querying.
  • Future directions: Future work includes testing AudioCLIP on a wider variety of datasets and tasks and using more powerful image and audio backbones.
Loading 2106.13043v1…