PaperlayerView exact revision on arXiv

Source-linked AI summary

Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation

Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Marianna Nezhurina, Taylor Berg-Kirkpatrick, Shlomo Dubnov

arXiv:2211.06687v4cs.SDeess.AS

TL;DR

The paper addresses the challenge of learning effective audio representations with limited finely annotated data. It builds a contrastive language-audio pretraining pipeline using large-scale audio-text data, encoder selection, feature fusion, and keyword-to-caption augmentation, achieving strong retrieval and classification results.

  • Problem

    Limited finely annotated audio data makes effective audio representation learning across tasks without extensive supervision challenging.

  • Method

    The paper releases LAION-Audio-630K and constructs a contrastive language-audio pretraining pipeline with multiple audio and text encoders, feature fusion, and keyword-to-caption augmentation.

  • Results

    The model outperforms previous methods on most text-to-audio retrieval metrics and achieves state-of-the-art zero-shot classification with supervised results comparable to existing models.

  • Takeaways & Limitations

    The proposed dataset and model are publicly available, and the experiments support their use for audio retrieval and classification representation learning.

Abstract

from arXiv · show

Contrastive learning has shown remarkable success in the field of multimodal representation learning. In this paper, we propose a pipeline of contrastive language-audio pretraining to develop an audio representation by combining audio data with natural language descriptions. To accomplish this target, we first release LAION-Audio-630K, a large collection of 633,526 audio-text pairs from different data sources. Second, we construct a contrastive language-audio pretraining model by considering different audio encoders and text encoders. We incorporate the feature fusion mechanism and keyword-to-caption augmentation into the model design to further enable the model to process audio inputs of variable lengths and enhance the performance. Third, we perform comprehensive experiments to evaluate our model across three tasks: text-to-audio retrieval, zero-shot audio classification, and supervised audio classification. The results demonstrate that our model achieves superior performance in text-to-audio retrieval task. In audio classification tasks, the model achieves state-of-the-art performance in the zero-shot setting and is able to obtain performance comparable to models' results in the non-zero-shot setting. LAION-Audio-630K and the proposed model are both available to the public.

1. INTRODUCTION

The paper addresses limited supervision for audio representation learning by combining audio with natural-language descriptions through contrastive pretraining. It releases a large dataset and introduces model and evaluation improvements for retrieval and classification.

  • Limited finely annotated audio data makes effective audio representation learning without extensive supervision challenging.
  • Contrastive language-audio learning can use paired audio-text data to capture crossmodal information without requiring task-specific annotation.
  • Prior studies were limited by relatively small datasets, incomplete encoder and hyperparameter investigations, variable-length audio handling, and narrow retrieval-focused evaluation.
  • The paper releases LAION-Audio-630K, a public dataset of 633,526 audio-text pairs, with keyword-to-caption augmentation for AudioSet labels.
  • The proposed pretraining pipeline tests two audio encoders and three text encoders and uses feature fusion to handle variable-length inputs.
  • Comprehensive experiments cover text-to-audio retrieval, zero-shot audio classification, and supervised audio classification, with reported state-of-the-art or comparable performance.
  • Both LAION-Audio-630K and the proposed model are publicly available.

2. LAION-AUDIO-630K AND TRAINING DATASET

LAION-Audio-630K is a large public audio-text dataset assembled from multiple sources, and the training setup combines datasets of different scales while expanding labels into captions.

  • LAION-Audio-630K contains 633,526 audio-text pairs totaling 4,325.39 hours from eight publicly available data sources.
  • The dataset includes human activities, natural sounds, and audio effects, making it broader than a single audio domain.
  • Training settings range from about 55K AudioCaps+Clotho pairs to around 630K LAION-Audio-630K pairs and 1.9 million AudioSet samples.
  • For label-only datasets, labels are converted into captions using templates or a keyword-to-caption model, increasing the combined captioned training data to 2.5 million audio samples.

3. MODEL ARCHITECTURE

The model separately encodes audio and text, projects them into a shared representation space, and trains them with bidirectional contrastive learning. It supports retrieval and classification while using feature fusion for variable-length audio and keyword-to-caption augmentation.

  • Contrastive Language-Audio Pretraining: Separate audio and text encoders process paired inputs before projection layers map their outputs into the same representation dimension.The audio and text embeddings are produced independently and projected with two-layer MLPs using ReLU activation.
  • Contrastive Language-Audio Pretraining: Bidirectional contrastive learning trains paired audio and text embeddings to align, using audio-to-text and text-to-audio logits with a learnable temperature.The loss operates over batches, where N denotes batch size during training and τ scales the logits.
  • Downstream Tasks in Inference Stage: Text-to-audio retrieval matches an audio embedding to the nearest text embedding by cosine similarity, while zero-shot classification compares audio with prompt-text embeddings.Supervised classification instead adds a projection layer and fine-tunes the model for fixed categories.
  • Feature Fusion for Variable-Length Audio: Feature fusion combines coarse global and randomly sampled local audio information so training and inference can use variable-length inputs in constant computation time.For longer audio, the method downsamples a global input and samples local clips; local features are then combined through another 2D-convolution layer.
  • Keyword-to-Caption Augmentation: Keyword-to-caption augmentation uses pretrained T5 to generate captions from audio keywords and applies gender de-biasing during post-processing.The generated captions expand textual supervision for audio-text contrastive pretraining.

4. EXPERIMENTS

The experiments evaluate encoder choices, dataset scaling, retrieval, and zero-shot and supervised classification. Feature fusion and keyword-to-caption augmentation support variable-length inputs and improve performance, while the best model achieves strong retrieval and classification results.

  • Experimental design: The study evaluates encoder combinations, dataset scale, feature fusion, keyword-to-caption augmentation, retrieval, and audio classification.Retrieval uses recall and mAP, while classification includes zero-shot and supervised settings.
  • Encoder selection: HTSAT with RoBERTa is selected as the best encoder combination because HTSAT outperforms PANN and RoBERTa outperforms BERT.The CLIP transformer performs worst, with high over-fitting and low generalization performance.
  • Text-to-audio retrieval: 36.7% R@1 on AudioCaps and 18.2% R@1 on Clotho are reported for text-to-audio retrieval, outperforming previous methods on most metrics.The best model uses large-scale training data, keyword-to-caption augmentation, and feature fusion.
  • Model improvements: Feature fusion enables variable-length audio inputs and improves performance, while keyword-to-caption augmentation substantially increases VGGSound and US8K results.The augmentation enriches the text embedding space with additional captions.
  • Audio classification: The models achieve new state-of-the-art zero-shot classification results across ESC-50, VGGSound, and US8K.The best supervised model surpasses the state of the art on VGGSound and approaches it on FSD50K.

5. CONCLUSION AND FUTURE WORK

The paper concludes that a large-scale audio-text dataset and improvements to language-audio contrastive learning improve audio understanding and task performance. It also identifies broader downstream applications as future work.

  • LAION-Audio-630K, keyword-to-caption augmentation for AudioSet, and feature fusion improve audio understanding and task performance.The approach also enables learning from variable-length audio data.
  • Future work: Future work includes collecting larger training datasets and applying the learned representations to audio synthesis and separation.
  • Retrieval evaluation: The retrieval evaluation uses R@1, R@5, R@10, and mAP, with special handling for datasets containing five text references per audio.Audio-to-text recall takes the best result among the five ground-truth captions, while text-to-audio metrics average over independent text samples.
  • Retrieval evaluation: For datasets with one text per audio, recall and mAP are measured using the standard evaluation procedure.

D. DETAILS OF LAION-AUDIO-630K

The paper documents LAION-Audio-630K and its supporting dataset materials, including source specifications, training-dataset details, and evidence of variable audio lengths. These materials describe why variable-length processing matters for the collected audio.

  • LAION-Audio-630K is documented through source specifications and dataset details in Tables 5 and 6.
  • The audio-length distribution of Epidemic Sound and Freesound demonstrates the variable-length problem in audio processing and model training.

D.1. Freesound Dataset

The Freesound subset consists of audio clips collected from Freesound, with licensing conditions determined individually by each uploader. The release documents the licenses associated with the included samples.

  • Freesound samples are collected from Freesound and released under Creative Commons licenses.Each clip has its own license as defined by its uploader.
  • Some included licenses require attribution, while others forbid further commercial reuse.
  • The dataset release page lists the license for each sample.

E. ATTENTIONAL FEATURE FUSION

The attentional feature fusion architecture combines global and merged local audio information through a two-branch CNN mechanism. The branches generate coefficients that determine how the two inputs are combined.

  • Attentional feature fusion uses a two-branch CNN network to combine global and local information from input audio.The architecture is designed to combine coarse global information with merged local information.
  • The architecture accepts X as global information and Y as merged local information.
  • Two CNN networks generate coefficients that are used to add the global and local inputs together.

F. ADDITIONAL EXPERIMENT OF FEATURE FUSION ON FREESOUND DATASET

Experiments on Freesound show that feature fusion improves text-to-audio retrieval performance on variable-length audio. Training with AudioCaps, Clotho, and LAION-Audio performs better than also adding AudioSet on this evaluation set.

  • Freesound contains more-than-10-second audio samples, making it an additional evaluation set for feature fusion.Its sample length is described as similar to the Clotho dataset.
  • AudioCaps + Clotho + LA. outperforms AudioCaps + Clotho + LA. + AudioSet on the Freesound evaluation set.The authors attribute the decrease to AudioSet shifting the model distribution toward AudioSet-like audio.
  • Feature fusion improves text-to-audio performance on Freesound’s variable-length audio samples.The result supports the paper’s conclusion that fusion produces better audio representations for variable-length inputs.
  • Table 7 reports text-to-audio retrieval performance on the Freesound evaluation set.

G. EXAMPLES OF KEYWORD-TO-CAPTION AUGMENTATION

The paper augments AudioSet labels by generating captions with T5 and uses a de-biased version for model training. Samples shorter than two seconds are excluded because their captions match poorly with the audio.

  • Keyword-to-caption augmentation uses a T5 model to generate captions from AudioSet labels.The paper also presents a de-biased version for model training.
  • Samples shorter than 2 seconds are excluded because they are often single events that match generated captions poorly.
  • For training with AudioSet, the authors use generated keyword-to-caption captions and exclude template-generated captions.
  • The captions are generated using the T5 implementation provided by keytotext.

H. EXPERIMENT SETTINGS ON DATA EXCLUSION

The zero-shot evaluation excludes all samples overlapping with the training data. Table 8 documents the overlaps removed before computing audio-classification metrics.

  • All overlaps between training data and zero-shot evaluation data are excluded before evaluation.
  • The remaining dataset is used to calculate the audio-classification metrics.
Loading 2211.06687v4…