PaperlayerView exact revision on arXiv

Source-linked AI summary

Natural Language Supervision for General-Purpose Audio Representations

Benjamin Elizalde, Soham Deshmukh, Huaming Wang

arXiv:2309.05767v2cs.SDeess.AS

TL;DR

Prior audio representations in the described benchmark setting required task-specific fine-tuning for downstream use. This paper trains CLAP with new audio and text encoders and 4.6 million audio-text pairs, then evaluates it across 26 downstream tasks. The authors report new zero-shot state of the art on most of those tasks.

  • Problem

    Prior audio representations in the described benchmark setting required task-specific fine-tuning for downstream use.

    Source: §1 · p. 1

  • Method

    CLAP combines an audio encoder trained on 22 tasks and GPT2 sentence representations with 4.6 million audio-text pairs, jointly aligning audio and text representations.

    Source: §1 · p. 1, §2.2 · p. 2

  • Results

    CLAP established new zero-shot state of the art on most of 26 downstream tasks.

    Source: §4.2 · p. 4, §5 · p. 5

  • Takeaways & Limitations

    The authors argue zero-shot audio models should be evaluated across different tasks with a focus on generalization rather than overfitting to specific tasks.

    Source: §5 · p. 5

    Audio-text retrieval on AudioCaps remained below the benchmark for audio-to-text, and both retrieval datasets were sensitive to out-of-domain training data.

    Source: §4.3 · p. 4

Abstract

from arXiv · show

Audio-Language models jointly learn multimodal text and audio representations that enable Zero-Shot inference. Models rely on the encoders to create powerful representations of the input and generalize to multiple tasks ranging from sounds, music, and speech. Although models have achieved remarkable performance, there is still a performance gap with task-specific models. In this paper, we propose a Contrastive Language-Audio Pretraining model that is pretrained with a diverse collection of 4.6M audio-text pairs employing two innovative encoders for Zero-Shot inference. To learn audio representations, we trained an audio encoder on 22 audio tasks, instead of the standard training of sound event classification. To learn language representations, we trained an autoregressive decoder-only model instead of the standard encoder-only models. Then, the audio and language representations are brought into a joint multimodal space using Contrastive Learning. We used our encoders to improve the downstream performance by a margin. We extensively evaluated the generalization of our representations on 26 downstream tasks, the largest in the literature. Our model achieves state of the art results in several tasks leading the way towards general-purpose audio representations.

1. INTRODUCTION

Prior audio representations are often fine-tuned for downstream tasks, while zero-shot models offer direct application across tasks. The paper proposes CLAP with new encoders and 4.6 million audio-text pairs, evaluating generalization on 26 downstream tasks.

1. INTRODUCTION

  • Prior audio representations were pretrained on AudioSet, and all methods in the described benchmark setup required task-specific fine-tuning for downstream use.The HEAR benchmark provided a comprehensive setup for comparing audio representations.Source: §1 · p. 1
  • The choice of audio and text encoders is critical to representation quality; prior work upgraded CNN to HTSAT and BERT to RoBERTa [4] [6].An earlier CLAP model achieved state of the art in 16 downstream tasks.Source: §1 · p. 1
  • Scaling training pairs can improve overall performance but may reduce it in some domains, making pair diversity and noise important considerations [4] [6].Prior systems used class labels or descriptions and reported gains on differing numbers of tasks, while adding music and speech pairs to LAION CLAP coincided with a 1% absolute ESC50 decline.Source: §1 · p. 1
  • The proposed CLAP uses an audio encoder trained on 22 audio tasks, an autoregressive decoder-only language model, and 4.6 million audio-text pairs; it is evaluated on 26 downstream tasks.The authors report state-of-the-art results in several tasks.Source: §1 · p. 1

2. METHOD

CLAP projects audio and text representations into a shared multimodal space and trains them with a symmetric contrastive objective. Its encoders are HTSAT-22 and GPT2, and the pretrained representations support zero-shot classification, retrieval, and audio captioning.

2.1. Contrastive Language-Audio Pretraining

  • Audio and text representations are projected into a shared multimodal space, compared by similarity, and jointly trained with a symmetric cross-entropy loss.The similarity matrix has matching pairs on its diagonal and non-matching pairs off-diagonal.Source: §2.1 · p. 2

2.2. Audio and Text Encoders

  • HTSAT-22 is a transformer-based audio encoder trained on 22 tasks, using an audio prompt and mapper to produce free-form text outputs autoregressively.Its training tasks include classification, captioning, retrieval, and audio question answering.Source: §2.2 · p. 2
  • GPT2 provides sentence-level text representations by using the final < |endoftext| > token’s representation during contrastive pretraining.The method uses this token to aggregate information from the input text.Source: §2.2 · p. 2

2.3. Evaluation

  • For zero-shot classification and retrieval, CLAP compares test-audio embeddings with text embeddings for class labels or queries; captioning routes audio through a mapper to GPT2.Captioning training updates only the mapper weights using a captioning loss and the training split.Source: §2.3 · p. 2
  • Table 1 compares zero-shot performance across 16 downstream tasks using the average of task metrics as its Zero-Shot score.The table uses 119k training pairs for the encoder comparison.Source: Table 1 · p. 3

3. EXPERIMENTS

Experiments use a 4.6-million-pair audio-text collection and evaluate 26 downstream tasks across several audio domains. The paper also specifies audio preprocessing, encoder comparisons, and the training schedule.

3. EXPERIMENTS

  • The 4.6-million-pair collection spans human and environmental sounds, acoustic scenes, music, sound effects, and speech; evaluation covers 26 tasks across multiple domains.The encoder comparison uses a subset of 16 tasks, and the authors excluded AudioCaps and Clotho test sets from training.Source: §3 · p. 3
  • Audio is represented as log Mel spectrograms sampled at 44.1 kHz, with 320-frame hops, 1024-frame windows, and 64 Mel bins over 50–8000 Hz.Training clips are randomly truncated to seven seconds or padded when shorter.Source: §3 · p. 3
  • Both encoders were unfrozen for 40 epochs with batch size 1,536, Adam, and an initial learning rate of 10^-3; reported results use the epoch with the best average zero-shot score.Overall performance peaked in the first 10 epochs; the temperature parameter was learnable and initialized to 0.007.Source: §3 · p. 3

4. RESULTS AND DISCUSSION

The proposed CLAP model outperformed benchmark systems on most of 26 downstream tasks, while encoder and training-pair results varied across tasks. Retrieval remained a weak area, particularly AudioCaps audio-to-text.

4.1. Proposed audio and text encoder

  • HTSAT-22+GPT2 scored 0.480 Zero-Shot score, an absolute 9% higher than HTSAT+RoBERTa (0.431) and CNN14+BERT (0.428).All encoder combinations performed better than random, although different combinations led on different tasks.Source: §4.1 · pp. 3–4

4.2. Scaling proposed CLAP architecture

  • The model set zero-shot state of the art on most of 26 tasks, outperforming four SoTA models, though some evaluations included training audio.For example, FSD50k audio and web descriptions were used in training, but not class labels.Source: Table 2 · p. 4, §4.2 · p. 4

4.1. Proposed audio and text encoder

  • Comparing HTSAT-22+GPT2 with HTSAT+GPT2 showed gains of 10% on LibriCount10, 7% on NS Instrument, and 6% on ESC50.The authors identify HTSAT-22, pretrained on 22 audio tasks rather than only sound-event classification, as the major contributor to performance improvement.Source: §4.1 · p. 4
  • GPT2 with either HTSAT or HTSAT-22 performed best among the text encoders; the authors hypothesize vocabulary size and end-token prediction emphasize word order.GPT2 predicts tokens through the end-of-text token to produce a sentence-level representation.Source: §4.1 · p. 4

4.3. Generalization and individual domain performance

  • Emotion-related pairs raised RAVDESS accuracy to 31.5% and CREMAD to 30%, while the 4.6M-pair model trailed smaller-pair systems on ESC50 and GTZAN.The authors conclude that performance varies across tasks as training pairs are added, and recommend evaluating generalization across domains and tasks.Source: §4.3 · p. 4
  • Clotho mAP@10 improved (A-T 0.155 vs 0.138; T-A 0.257 vs 0.204), but AudioCaps A-T lagged (0.319 vs 0.457).Both retrieval datasets are sensitive to out-of-domain training data, and adding training pairs did not translate into improvement.Source: §4.3 · p. 4

5. CONCLUSION

The authors introduce a CLAP model with proposed encoders and 4.6M training pairs, then evaluate generalization across 26 tasks. They report SoTA on most tasks and argue evaluation should prioritize generalization over task-specific overfitting.

  • SoTA results on most of 26 evaluated tasks support the model’s generalization across tasks.Source: §5 · p. 5
  • The model combines the authors’ proposed encoders with 4.6M training pairs.Source: §5 · p. 5
  • The authors argue zero-shot models should be evaluated across tasks with a focus on generalization rather than overfitting to specific tasks.Source: §5 · p. 5
Loading 2309.05767v2…