Source-linked AI summary
Contrastive Learning of General-Purpose Audio Representations
Aaqib Saeed, David Grangier, Neil Zeghidour
TL;DR
Audio representation learning needs a general-purpose self-supervised approach beyond speech-focused methods. COLA uses contrastive pre-training with same-clip positives and different-clip negatives, and its AudioSet-trained embeddings transfer across nine diverse tasks, outperforming prior unsupervised systems and supervised baselines after fine-tuning.
Problem
Prior audio representation-learning work largely focuses on speech tasks, motivating a general-purpose approach covering broader sound domains.
Method
COLA pre-trains a convolutional audio encoder contrastively, pairing segments from the same clip as positives and segments from different clips in the batch as negatives.
Results
COLA embeddings outperform previous unsupervised methods across most downstream tasks, with fine-tuning exceeding supervised in-domain baselines across the evaluated tasks.
Takeaways & Limitations
COLA provides a simple, easy-to-implement framework for learning transferable audio representations across speech, music, acoustic scenes, and animal sounds.
Abstract
from arXiv · showhide
We introduce COLA, a self-supervised pre-training approach for learning a general-purpose representation of audio. Our approach is based on contrastive learning: it learns a representation which assigns high similarity to audio segments extracted from the same recording while assigning lower similarity to segments from different recordings. We build on top of recent advances in contrastive learning for computer vision and reinforcement learning to design a lightweight, easy-to-implement self-supervised model of audio. We pre-train embeddings on the large-scale Audioset database and transfer these representations to 9 diverse classification tasks, including speech, music, animal sounds, and acoustic scenes. We show that despite its simplicity, our method significantly outperforms previous self-supervised systems. We furthermore conduct ablation studies to identify key design choices and release a library to pre-train and fine-tune COLA models.
1. INTRODUCTION
COLA addresses the limited focus of prior audio representation learning on speech by proposing a simple contrastive framework for general-purpose sound representations. Pre-training on AudioSet transfers effectively across diverse tasks, with frozen embeddings outperforming random representations and fine-tuning exceeding supervised in-domain performance.
- Motivation: Prior audio representation learning largely focused on speech tasks despite progress from triplet-loss and temporal-prediction approaches.Earlier methods included metric learning, speech benchmarks, and AUDIO2VEC-style temporal-distance prediction.
- COLA: COLA learns general-purpose sound representations by contrasting segments from the same clip against segments from different clips.The framework builds on contrastive learning advances in computer vision and reinforcement learning.
- COLA: The method avoids augmentation strategies and memory banks while using many in-batch negatives for each positive pair.Same-clip sampling creates positives, while different clips in the batch provide dissimilar pairs.
- Evaluation: COLA is evaluated after AudioSet pre-training across diverse downstream domains, including speech, music, acoustic scenes, and animal sounds.The evaluation covers nine downstream classification tasks.
- Results: Fine-tuned COLA exceeds a fully supervised in-domain convolutional network, while the approach outperforms previous unsupervised systems on most downstream tasks.The authors present COLA as simple and easy to implement for general-purpose audio representation learning.
2. METHOD
COLA pre-trains an audio encoder with contrastive learning, using same-clip segments as positives and different-clip batch segments as negatives. Its bilinear similarity and multi-distractor loss support efficient representation learning from unlabeled audio.
- Contrastive objective: COLA pre-trains a convolutional feature extractor by maximizing agreement between segments from the same clip and contrasting them with segments from different clips.The resulting extractor is paired with a classification layer for downstream audio understanding tasks.
- Representation and similarity: The encoder maps log-compressed mel-filterbanks x into a latent representation h that transfers to downstream classification.A shallow network then maps h into z for bilinear comparisons.
- Representation and similarity: Bilinear similarity compares the projected representations rather than using the more common cosine similarity.The paper later evaluates this design choice through ablation.
- Design choices: Bilinear similarity outperforms cosine similarity on all downstream tasks, establishing it as the default similarity method in the paper.The corresponding comparison appears in the similarity-measure ablation.
- Contrastive objective: The multi-class cross-entropy loss uses one same-clip positive and a set of negative distractors, leveraging multiple distractors at once.This differs from triplet loss, which uses fewer comparisons per objective.
- Batch construction: Batch positives serve as negatives for other anchors, making each example participate in all batch comparisons without maintaining a memory bank.The study varies batch sizes from 256 to 2048.
3. EXPERIMENTS
COLA is evaluated by pre-training on unlabeled AudioSet and transferring one audio embedding across nine diverse downstream datasets. It consistently outperforms random, supervised-from-scratch, and prior self-supervised baselines, while ablations identify bilinear similarity and a batch size of 1024 as effective choices.
- Evaluation setup: COLA embeddings are pre-trained on unlabeled AudioSet and transferred using either a frozen linear classifier or end-task fine-tuning across diverse audio datasets.The evaluation targets transferability across speech and non-speech domains and recording conditions.
- Results: 74.3% against 29.1% on average shows that a linear classifier on frozen COLA embeddings outperforms a classifier trained on random embeddings.The result is reported across nine downstream datasets.
- Results: 85.1% against 83.9% on average shows that COLA initialization improves fine-tuned supervised performance on all tasks but language identification.COLA is used as initialization for one fine-tuned model per downstream task.
- Results: COLA embeddings consistently outperform the compared triplet-loss, AUDIO2VEC, and temporal-gap self-supervised methods on the evaluated tasks.On acoustic scene classification, COLA reaches 94% versus 73% with triplet loss, and it improves speech-command and musical-instrument classification by 30% absolute on both tasks.
- Ablations: Bilinear similarity produces the best downstream results in all evaluated cases, outperforming cosine similarity in the ablation study.The cosine comparison uses temperature τ = 0.2 before loss computation.
- Ablations: A batch size of 1024 provides better average representations than smaller sizes, whereas increasing it to 2048 worsens performance in most cases.Larger batches provide more negative samples, but the largest tested batch is not optimal.
4. CONCLUSIONS
COLA is presented as a simple, easy-to-implement self-supervised contrastive algorithm for general-purpose audio representation learning. It improves over earlier unsupervised methods in linear evaluation and over supervised baselines through fine-tuning across varied downstream tasks.
- 4. CONCLUSIONS: COLA combines simple implementation with strong transferability across audio tasks, supporting its use as a baseline for future self-supervised audio work.The conclusion describes this as the authors’ expected role for the system.