Source-linked AI summary
A Framework for Generative and Contrastive Learning of Audio Representations
Prateek Verma, Julius Smith
TL;DR
The paper addresses audio representation learning without ground-truth labels, proposing contrastive learning from augmented signals and Transformer prediction over learned dictionary codes. The authors report considerable performance using learned representations with a linear classifier, including a 15% supervised–unsupervised gap in their experiments versus around 10% in [2].
Problem
The paper seeks audio representations learned without ground-truth labels, using data itself for supervision rather than labels to guide neural networks.
Source: §1 · p. 1Method
The paper learns representations by bringing augmented versions of the same audio together contrastively and by training Transformers to predict the next element in a learned dictionary.
Source: §1 · pp. 1–2Results
Takeaways & Limitations
The authors conclude that contrastive and Transformer-based representations trained on data can achieve considerable performance with a linear head, and may suit tasks where large-scale labeling is not feasible.
Source: §5 · p. 4
Abstract
from arXiv · showhide
In this paper, we present a framework for contrastive learning for audio representations, in a self supervised frame work without access to any ground truth labels. The core idea in self supervised contrastive learning is to map an audio signal and its various augmented versions (representative of salient aspects of audio like pitch, timbre etc.) to a space where they are close together, and are separated from other different signals. In addition we also explore generative models based on state of the art transformer based architectures for learning latent spaces for audio signals, without access to any labels. Here, we map audio signals on a smaller scale to discrete dictionary elements and train transformers to predict the next dictionary element. We only use data as a method of supervision, bypassing the need of labels needed to act as a supervision for training the deep neural networks. We then use a linear classifier head in order to evaluate the performance of our models, for both self supervised contrastive and generative transformer based representations that are learned. Our system achieves considerable performance, compared to a fully supervised method, with access to ground truth labels to train the neural network model. These representations, with avail-ability of large scale audio data show promise in various tasks for audio understanding tasks
1. INTRODUCTION AND RELATED WORK
Prior work learns audio representations through generative or discriminative approaches, often guided by labels, while self-supervised methods use data itself as supervision. This paper proposes contrastive learning from audio augmentations and a dictionary-based Transformer model for audio representations.
1. INTRODUCTION AND RELATED WORK
- Self-supervised learning uses data itself as supervision, building on masked prediction methods and the availability of large amounts of unlabeled data.Prior audio work and contrastive losses had shown promise, while BERT predicts masked entities for downstream tasks.Source: §1 · p. 1
- Earlier work includes generative and discriminative representation methods, with supervised approaches using labels to guide neural-network representations.Examples include SoundNet’s use of image latent spaces as a target for learning from waveforms.Source: §1 · p. 1
- Contrastive learning maps augmented versions of an audio signal together while separating them from augmented versions of other signals.The proposed signal-processing augmentations are designed to capture fundamental aspects of audio.Source: §1 · pp. 1–2
- A dictionary-based generative model uses Transformers to learn audio representations that can be evaluated on downstream tasks with a linear head.The model learns to predict the next element in a learned dictionary space.Source: §1 · pp. 1–2
2. DATASET AND AUDIO REPRESENTATION
The experiments use UrbanSound 8K and represent each excerpt as a downsampled log-magnitude mel-spectrogram.
2. DATASET AND AUDIO REPRESENTATION
- UrbanSound 8K contains 8,732 labeled excerpts of urban sounds, each under four seconds, across 10 classes.The experiments use the dataset’s supplied folds and metadata.Source: §2 · p. 2
- Excerpts shorter than four seconds are repeated to make all audio inputs four seconds long.Source: §2 · p. 2
- Each excerpt is converted to a 64 × 200 log-magnitude mel-spectrogram after downsampling the 128 × 400 representation by a factor of two.The original representation uses 16 kHz audio, a 10 ms hop, a 30 ms window, a 1024-point FFT, and 128 mel bins.Source: §2 · p. 2
3. METHOD
The method learns audio representations without annotated training labels through two approaches: contrastive learning on augmented spectrograms and Transformer prediction over discrete dictionary codes. The Transformer pipeline first converts spectrogram patches into cluster codes, then predicts the next code from prior context.
3. METHOD
- The method uses data itself for supervision, combining contrastive learning on feature-specific augmentations with future-step prediction as a pseudo-task.The future-prediction approach treats a feature vector as summarizing past audio content.Source: §3 · p. 2
3.1.1. Data-Augmentation
- Nine spectrogram augmentations include checkerboard noise, time reversal, flipping, amplitude scaling, band-limiting, peak retention, energy thresholding, random noise, and spectral-envelope extraction.The augmentations are intended to create varied, feature-specific representations.Source: §3.1.1 · pp. 2–3
3.1.2. Algorithm
- A VGG-16 encoder and two-layer MLP map paired augmentations into a space where same-signal pairs are pulled together against other batch examples.The setup uses a 512-dimensional encoder output, a 128-neuron MLP, and varies temperature and batch size.Source: §3.1.2 · p. 3
3.2. Generative learning using Transformers
- The generative approach combines k-means clustering with Transformer-based dependency learning to represent audio without labels.Source: §3.2 · p. 3
3.2.1. Latent space learning
- Four-frame spectrogram patches are encoded and clustered into discrete codes, yielding a 50-element code sequence with vocabularies of 16, 64, or 256 clusters.The patches are 64 × 4 inputs to an autoencoder with a 16-unit bottleneck; k-means is fit on training-set patches.Source: §3.2.1 · p. 3
3.2.2. Transformers on discrete latent spaces
- The Transformer predicts the next code from 25 prior codes and uses its output representation for classification with a linear head.The described setup uses a 256-code vocabulary, eight attention heads, and input corruption during 20% of training steps.Source: §3.2.2 · p. 4
4. RESULTS AND DISCUSSION
The experiments compare generative models with contrastive learning and a baseline. The authors report a comparable supervised–unsupervised performance gap and strong generative results with fewer parameters under nearly equal training-data conditions.
4. RESULTS AND DISCUSSION
- 15% in this work versus around 10% in [2] is the reported supervised–unsupervised performance gap.The authors describe the gaps as comparable.Source: §4 · p. 4
- The reported experiments follow a preliminary search over contrastive batch size and temperature and Transformer architecture and latent-code settings.The caption lists attention heads, feed-forward dimensions, and latent code among the Transformer settings searched.Source: Figure 3 · p. 4
- With almost the same amount of training data, generative models achieve strong results using a fraction of the parameters used by the contrastive approach.The comparison is described as a controlled experimental setting.Source: §4 · p. 4
5. CONCLUSION AND FUTURE WORK
The conclusion presents contrastive learning from augmented audio and Transformer-based next-element prediction as two approaches to audio representation learning. It reports considerable performance from data-trained representations evaluated with a linear head and points to applications where large-scale labeling is not feasible.
5. CONCLUSION AND FUTURE WORK
- The paper proposes contrastive learning based on data augmentation and Transformer-based next-element prediction for audio representation learning.The authors describe both approaches as self-supervised.Source: §5 · p. 4
- The learned representations achieve considerable performance when classified with a linear head after training the deep networks on data.The conclusion contrasts this evaluation with multilayer nonlinear heads trained with labels.Source: §5 · p. 4
- The authors identify applications with infeasible large-scale labeling and propose exploring different losses, encoders, clustering, and multitask learning.They also suggest combining the embeddings with other neural architectures in vision, speech, and natural language processing.Source: §5 · p. 4