Source-linked AI summary
Unsupervised Learning of Semantic Audio Representations
Aren Jansen, Manoj Plakal, Ratheet Pandya, Daniel P. W. Ellis, Shawn Hershey, Jiayang Liu, R. Channing Moore, Rif A. Saurous
TL;DR
Labeling nonspeech audio is costly, while unlabeled recordings contain semantic structure that could support learning sound categories. This paper uses semantic constraints to train triplet-loss audio embeddings without labels, which outperform raw inputs across retrieval and classification and yield strong limited-supervision results.
Problem
Manually labeling data for each new sound application is costly and time consuming, although unlabeled audio is easy to collect.
Source: §1 · p. 1Method
The paper samples triplets from unlabeled audio using category-preserving transformations, mixtures, and temporal proximity, then trains convolutional networks with triplet loss.
Source: §1 · p. 1, §3.2.3 · p. 2, §3.2.4 · p. 2Results
Unsupervised embeddings outperform raw inputs on retrieval and classification, and the fully unsupervised ResNet-50 embedding reaches 85% (0.244/0.288) of the fully supervised model's mAP for classification.
Source: §5 · p. 4, §4.4 · p. 4Takeaways & Limitations
With just 20 examples per class, a single-layer classifier on the unsupervised embedding doubles the mAP of a full ResNet-50 classifier trained from raw inputs.
Source: §4.4 · p. 4Temporal-proximity triplets rely on the assumption that events from the same recording, especially those close in time, are likely to have the same or related categories.
Source: §1 · p. 1, §3.2.5 · p. 3
Abstract
from arXiv · showhide
Even in the absence of any explicit semantic annotation, vast collections of audio recordings provide valuable information for learning the categorical structure of sounds. We consider several class-agnostic semantic constraints that apply to unlabeled nonspeech audio: (i) noise and translations in time do not change the underlying sound category, (ii) a mixture of two sound events inherits the categories of the constituents, and (iii) the categories of events in close temporal proximity are likely to be the same or related. Without labels to ground them, these constraints are incompatible with classification loss functions. However, they may still be leveraged to identify geometric inequalities needed for triplet loss-based training of convolutional neural networks. The result is low-dimensional embeddings of the input spectrograms that recover 41% and 84% of the performance of their fully-supervised counterparts when applied to downstream query-by-example sound retrieval and sound event classification tasks, respectively. Moreover, in limited-supervision settings, our unsupervised embeddings double the state-of-the-art classification performance.
1. INTRODUCTION
The paper addresses the cost of labeling nonspeech audio by using semantic properties that hold without category labels. It turns those properties into triplet-loss training constraints and evaluates the resulting embeddings on retrieval and classification.
1. INTRODUCTION
- Labeling each new sound application is costly and time consuming, despite unlabeled audio being easy to collect.Source: §1 · p. 1
- Noise and small time or frequency translations preserve event categories, mixtures can represent constituents, and nearby recording events are likely to be related.Source: §1 · p. 1
- Triplet loss can learn relative similarity constraints without category labels, making it suitable for these semantic relationships.A noisy event can be treated as more similar to its clean recording than to an arbitrary sound.Source: §1 · p. 1
- Unsupervised embeddings recover up to 84% of the performance gap between raw log mel inputs and fully supervised embeddings.The embeddings are evaluated on query-by-example sound retrieval and supervised sound event classification.Source: §1 · p. 1
2. RELATED WORK
Prior work includes unsupervised audio representations that do not explicitly elicit semantic structure, as well as related constraint-based learning in speech and vision. The paper situates its approach alongside these efforts.
2. RELATED WORK
- Earlier audio representation work used convolutional deep belief networks for speech and music and a denoising autoencoder variant for environmental sound classification, without explicitly eliciting semantic structure.Source: §2 · p. 1
- Speech work learns meaningful units from untranscribed speech using constraints from linguistic structure, including repeated words and shared conversation topics.Source: §2 · pp. 1–2
- Vision self-supervision uses constraints such as egomotion, spatial context, object tracking, and colorization, with extensions to joint audio-visual embeddings [24] [27].Source: §2 · p. 2
3. LEARNING ALGORITHM
The learning algorithm has two stages: sample triplets from unlabeled audio, then train a convolutional network to map spectrogram context windows into a low-dimensional space using triplet loss.
3. LEARNING ALGORITHM
- Training first samples triplets from collections of unlabeled audio recordings.Source: §3 · p. 2
- The model maps spectrogram context windows with F frequency channels and T frames into a lower d-dimensional vector space.Source: §3 · p. 2
- Triplet loss optimization trains convolutional neural networks to learn the mapping.Source: §3 · p. 2
3.1. Metric Learning with Triplet Loss
Triplet metric learning maps spectrogram inputs into a low-dimensional space where Euclidean distances express relative relationships among examples. Its loss encourages positives to be closer to anchors than negatives, while training uses semi-hard negative mining.
3.1. Metric Learning with Triplet Loss
- The map g sends spectrogram inputs from R^(F×T) to R^d, where simple Euclidean distance represents complex input-space relationships.Source: §3.1 · p. 2
- Each training triplet contains an anchor, a positive, and a negative example, and the loss pushes the positive closer to the anchor than the negative by margin δ.The loss is zero when all triplets satisfy the required inequality.Source: §3.1 · p. 2
- Triplets act as hard constraints that encode relative pairwise rankings, with learned distance serving as a proxy for those relationships.Source: §3.1 · p. 2
- Within-batch semi-hard negative mining selects the closest negative still farther from the anchor than the positive, avoiding negatives vulnerable to label noise.Source: §3.1 · p. 2
3.2. Triplet Sampling Methods
The section defines six ways to form triplets, from labeled category relationships to unlabeled transformations and temporal context. It also describes combining triplet sources into a joint embedding objective.
3.2.1. Explicitly Labeled Data
- With labeled data, anchor-positive pairs share a category and the negative comes from a different category, setting the experiments’ supervised performance topline.Source: §3.2.1 · p. 2
3.2.2. Gaussian Noise
- Gaussian-noise triplets pair an unlabeled example with a noise-perturbed version as anchor and positive, then use another example as the negative.The perturbation multiplies each time-frequency cell by 1 + |ε_tf|, with ε_tf drawn from a zero-mean Gaussian distribution.Source: §3.2.2 · p. 2
3.2.3. Time and Frequency Translation
- Time and frequency translations form positives by shifting an input spectrogram in time and frequency, then pairing it with another example as the negative.The time shift is circular; the frequency shift is truncated, with missing values set to zero energy.Source: §3.2.3 · p. 2
3.2.4. Example Mixing
- Example mixing forms a positive by combining the anchor and negative, avoiding an unsupported assumption about which random negative should be farther from a mixed sound.The positive is xa + α[E(xa)/E(xn)]xn, with α as a hyperparameter.Source: §3.2.4 · pp. 2–3
3.2.5. Temporal Proximity
- Temporal proximity sampling pairs events from the same recording as anchor and positive, while drawing the negative from a different recording.The positive pair can be restricted to start times less than Δt apart; with overlapping context windows and sufficiently small Δt, this is functionally similar to time translation.Source: §3.2.5 · p. 3
3.2.6. Joint Training
- Triplet sets from the sampling methods can be mixed to train a joint embedding, with source importance adjustable through loss weights or sample sizes.This is presented as an alternative to multi-task objectives when combining domains with distinct label inventories is difficult.Source: §3.2.6 · p. 3
4. EXPERIMENTS
The experiments evaluate unsupervised triplet embeddings on AudioSet retrieval and sound classification, with comparisons to raw features and supervised models. Joint unsupervised embeddings improve both tasks and also benefit classification with limited labels.
4. EXPERIMENTS
- The evaluation covers query-by-example retrieval without further training, shallow supervised classification, and lightly supervised classification; AudioSet supplies the manually annotated sound events.AudioSet contains over 2 million 10-second segments.Source: §4 · p. 3, §4.1 · p. 3
4.2. Model Architecture
- The ResNet-50 models produce length-normalized 128-dimensional embeddings from 64×96 context windows, a 48-fold reduction from the input dimensionality.Length normalization makes squared Euclidean distance proportional to cosine distance.Source: §4.2 · p. 3
4.3. Query-by-Example Retrieval
- Larger frequency shifts substantially improve translation-triplet retrieval over time translations alone; Gaussian-noise and example-mixing results depend relatively weakly on their sampling settings.The selected validation settings were also optimal on evaluation; proximity was not further limited beyond the AudioSet clip duration.Source: Table 1 · p. 3, §4.3 · p. 4
- Joint unsupervised embeddings improve retrieval mAP by 15% absolute over input spectrograms and recover more than 40% of the baseline-to-topline gap.The comparison is reported for the separate evaluation set.Source: §4.3 · p. 4, Table 2 · p. 4
4.4. Sound Classification
- 85% (0.244/0.288) mAP is achieved by the unsupervised ResNet-50 embedding relative to the fully supervised embedding with a single hidden-layer classifier.Combining triplet sets provides additional gains, and temporal proximity is the standout method.Source: §4.4 · p. 4, Table 2 · p. 4
- With 20 labeled examples per class, a single-layer classifier on the joint unsupervised embedding doubles the mAP of a ResNet-50 classifier trained on raw inputs.The 20 examples per class were used in each of three randomly sampled training trials.Source: §4.4 · p. 4, Table 3 · p. 4
5. CONCLUSIONS
The paper learns audio representations by sampling triplets under audio-specific semantic constraints that do not require labels. The resulting embeddings outperform raw inputs on retrieval and classification, with complementary constraints and benefits in limited-supervision settings.
5. CONCLUSIONS
- Audio-specific semantic constraints enable unsupervised triplet sampling that produces representations outperforming raw inputs on sound event retrieval and classification.The different constraints are complementary, and combining them improves the joint triplet-loss embedding.Source: §5 · p. 4
- The best unsupervised embedding provides an advantage when training sound event classifiers with limited supervision.Source: §5 · p. 4