Source-linked AI summary
Learning Contextual Tag Embeddings for Cross-Modal Alignment of Audio and Tags
Xavier Favory, Konstantinos Drossos, Tuomas Virtanen, Xavier Serra
TL;DR
The paper addresses the inability of fixed tag representations in prior cross-modal audio methods to generalize to unseen terms. It aligns audio embeddings with contextualized tag embeddings from pretrained word representations using contrastive learning. Self-attention benefits sound-event and music-genre classification, though benefits are not observed for musical-instrument classification.
Problem
Prior cross-modal audio–tag methods use fixed tag inputs that cannot generalize to terms unseen during training.
Source: §1 · p. 1Method
The method aligns audio-encoder embeddings with contextualized tag embeddings from pretrained word representations using contrastive learning, while reconstructing audio from the audio embeddings.
Source: §1 · p. 1, §2.1 · p. 2, §2.3 · p. 3Results
Self-attention benefits sound-event and music-genre classification, with more attention heads improving performance there, but no benefit is observed for musical-instrument classification or multiple heads on US8K sound-event recognition.
Source: §4 · p. 4Takeaways & Limitations
The learned audio encoder is evaluated as a reusable embedding extractor across sound-event, music-genre, and music-instrument classification tasks.
Source: §3 · p. 3, §5 · p. 4The benefit of self-attention is task-dependent: it is absent for musical-instrument classification, and multiple heads add no benefit on US8K sound-event recognition.
Source: §4 · p. 4
Abstract
from arXiv · showhide
Self-supervised audio representation learning offers an attractive alternative for obtaining generic audio embeddings, capable to be employed into various downstream tasks. Published approaches that consider both audio and words/tags associated with audio do not employ text processing models that are capable to generalize to tags unknown during training. In this work we propose a method for learning audio representations using an audio autoencoder (AAE), a general word embeddings model (WEM), and a multi-head self-attention (MHA) mechanism. MHA attends on the output of the WEM, providing a contextualized representation of the tags associated with the audio, and we align the output of MHA with the output of the encoder of AAE using a contrastive loss. We jointly optimize AAE and MHA and we evaluate the audio representations (i.e. the output of the encoder of AAE) by utilizing them in three different downstream tasks, namely sound, music genre, and music instrument classification. Our results show that employing multi-head self-attention with multiple heads in the tag-based network can induce better learned audio representations.
1. INTRODUCTION
The introduction motivates self-supervised audio representations and identifies a limitation in cross-modal methods that use fixed tag inputs. It proposes pretrained word embeddings and attention to form contextualized tag representations for audio–tag alignment.
1. INTRODUCTION
- Supervised audio representation learning relies on large annotated datasets that require considerable effort to build and remain limited in size.The introduction presents unsupervised, self-supervised, and semi-supervised approaches as alternatives for learning generic representations for downstream tasks.Source: §1 · p. 1
- Contrastive learning uses positive and negative pairs to learn latent spaces reflecting semantic characteristics, while prior machine-listening approaches described here consider only audio.The introduction also notes image–text embedding approaches that use self-attention for context-sensitive text embeddings.Source: §1 · p. 1
- Fixed tag inputs in COALA cannot generalize to terms unseen during training, limiting the flexibility of cross-modal contrastive representation learning.The proposed approach instead uses pretrained word embeddings to support textual generalization.Source: §1 · p. 1
- The proposed method uses pretrained word embeddings and attention to learn higher-level contextualized semantic representations from tags accompanying audio.It uses tags rather than text and describes a simpler approach for computing semantic embeddings.Source: §1 · pp. 1–2
2. PROPOSED METHOD
The method learns audio embeddings by jointly aligning audio and tag representations while reconstructing the input spectrogram. It uses convolutional audio encoding and decoding alongside multi-head self-attention over pre-trained tag embeddings.
2.1. Audio encoding and decoding
- A convolutional audio encoder processes the spectrogram through cascaded CNNs, batch normalization, ReLU, and dropout before flattening and layer normalization produce the audio embedding.That embedding is used both for cross-modal alignment and as the decoder input.Source: §2.1 · p. 2
2.3. Cross-modal alignment and optimization
- The audio embedding is aligned with the associated tags using contrastive loss and also feeds the decoder to reconstruct the spectrogram, combining semantic and low-level acoustic information.The objective combines reconstruction loss with contrastive loss between paired audio and tag representations and other minibatch examples.Source: §2.1 · p. 2, §2 · p. 2, §2.3 · p. 3
2.1. Audio encoding and decoding
- The audio decoder processes the embedding in reverse through transposed 2D CNNs and produces a reconstructed spectrogram.The reconstruction is obtained by applying a sigmoid to the decoder output.Source: §2.1 · p. 2
2.2. Multi-head, self-attention tags encoding
- Multi-head self-attention contextualizes pre-trained embeddings of the unordered tag set, without positional embeddings, to create the tag representation for contrastive alignment.Attention-head outputs are concatenated, then processed with a feed-forward network and layer normalization.Source: §2.2 · pp. 2–3
3. EVALUATION
The evaluation tests the audio encoder as a pretrained embedding extractor against COALA and MFCC features across sound, genre, and instrument classification. It uses Freesound audio-tag pairs for pretraining and separate datasets and protocols for each downstream task.
3.2. Audio-based classification
- Audio embeddings are evaluated against COALA and MFCC features in sound event recognition, music genre classification, and musical instrument classification.The tasks use UrbanSound8K, fault-filtered GTZAN, and a balanced 20,000-sample NSynth training subset, respectively.Source: §3 · p. 3, §3.2 · p. 4
3.1. Pre-training dataset and data pre-processing
- Pretraining uses Freesound audio-tag pairs, with processed tags supplying the word-embedding model.The tag processing retains the 1,000 most frequent tags after removing stop words, singularizing plurals, and excluding tags present in more than 70% of sounds.Source: §3.1 · p. 3
- The tag encoder varies word-embedding dimensionality and attention-head count, with mean aggregation as a comparison.The tested word-embedding dimensions are 128 and 1,152; attention setups use one or multiple heads.Source: §3.1 · p. 3
3.2. Audio-based classification
- Each downstream classifier is a one-hidden-layer MLP trained on the task-specific audio embeddings and evaluated using that task’s protocol.MLP training is repeated 10 times per task, and mean accuracy is reported.Source: §3.2 · pp. 3–4
4. RESULTS
The results indicate that tag-network self-attention benefits sound event recognition and music genre classification, but not musical instrument classification. Multiple attention heads help on the first two tasks, while tag-based US8K classification shows no benefit from multiple heads.
4. RESULTS
- Self-attention benefits SER and MGC, but no benefit is observed for MIC.The authors suggest instrument classification in the evaluated dataset may rely less on textual-description context.Source: Table 1 · p. 4, §4 · p. 4
- More attention heads improve SER and MGC, while the effect of word-embedding size remains unclear.Using different dimensions for the audio autoencoder and tag encoder does not necessarily hinder contrastive alignment.Source: §4 · p. 4
- For tag-based US8K classification, multiple heads provide no benefit, and audio-based classification performs below tag-based classification.Self-attention only slightly improves tag-based performance over mean aggregation.Source: §4 · p. 4
5. CONCLUSION
The paper presents cross-modal alignment of audio and tags using pretrained word embeddings and contextual tag embeddings. The resulting audio representations are evaluated across three classification tasks, and future work will test other scenarios.
5. CONCLUSION
- The method aligns contextual tag embeddings from pretrained word embeddings with audio embeddings using contrastive learning.The approach is intended to learn semantically enriched audio representations from audio samples and their associated tags.Source: §5 · p. 4
- The audio embeddings are evaluated for sound event recognition, music genre classification, and musical instrument classification.The paper states pretrained word embeddings offer broader application possibilities, including cross-modal retrieval and zero-shot learning.Source: §5 · p. 4
- Future work will focus on improving the model and evaluating it in different scenarios.The conclusion names cross-modal retrieval and zero-shot learning as example scenarios.Source: §5 · p. 4