Source-linked AI summary
MuLan: A Joint Embedding of Music Audio and Natural Language
Qingqing Huang, Aren Jansen, Joonseok Lee, Ravi Ganti, Judith Yue Li, Daniel P. W. Ellis
TL;DR
Existing music tagging and retrieval systems rely on rigid, predefined ontologies, motivating a direct link between audio and unconstrained musical language. MuLan trains a two-tower joint audio-text embedding on 44 million music recordings with weakly associated text, and demonstrates broad transfer, retrieval, tagging, and language-understanding capabilities.
Problem
Music tagging and retrieval traditionally use rigid predefined ontologies rather than unconstrained natural-language descriptions.
Method
MuLan uses cross-modal contrastive learning with separate audio and text towers to create a shared embedding space from weakly paired music audio and text.
Results
MuLan demonstrates versatile performance across transfer learning, cross-modal retrieval, zero-shot tagging, and music-domain language understanding, with state-of-the-art transfer learning on music tagging benchmarks.
Takeaways & Limitations
The learned representation provides a flexible natural-language interface to music audio while retaining general-purpose utility for music tagging.
Takeaways & Limitations
Simple negation remains difficult because the text encoder does not adequately model negated concepts.
Abstract
from arXiv · showhide
Music tagging and content-based retrieval systems have traditionally been constructed using pre-defined ontologies covering a rigid set of music attributes or text queries. This paper presents MuLan: a first attempt at a new generation of acoustic models that link music audio directly to unconstrained natural language music descriptions. MuLan takes the form of a two-tower, joint audio-text embedding model trained using 44 million music recordings (370K hours) and weakly-associated, free-form text annotations. Through its compatibility with a wide range of music genres and text styles (including conventional music tags), the resulting audio-text representation subsumes existing ontologies while graduating to true zero-shot functionalities. We demonstrate the versatility of the MuLan embeddings with a range of experiments including transfer learning, zero-shot music tagging, language understanding in the music domain, and cross-modal retrieval applications.
1. INTRODUCTION
MuLan addresses the limits of fixed music ontologies by jointly embedding music audio and unconstrained natural language. It uses weakly associated annotations from over 44 million music videos to support flexible music concepts and multiple downstream applications.
- Motivation: Fixed class inventories and manually specified ontologies restrict conventional content-labeling systems.Recent work has explored less restrictive natural-language interfaces for accessing categorical information in raw content signals.
- Motivation: Environmental-audio captioning has been constrained by scarce, small, and insufficiently diverse audio-text datasets, with weak zero-shot performance.Large-scale captioned image resources are more available, whereas environmental audio datasets do not span diverse sound-descriptive language.
- Data: Music metadata, comments, and playlists provide text annotations that are more likely to describe musical concepts because music is often the central focus.The mined annotations target concepts such as genres, artists, moods, and structure, although only a fraction truly refer to musical content.
- Contribution: MuLan trains a semantically structured music-audio embedding with a natural-language interface using a large-scale dataset of over 44 million internet music videos.The model supports transfer learning, cross-modal text-to-music retrieval, zero-shot music tagging, and music-domain language understanding.
2. RELATED WORK
Prior work spans task-agnostic audio representation learning, cross-modal contrastive alignment, and music-text modeling with fixed vocabularies or fused architectures. MuLan instead emphasizes weakly paired large-scale text supervision and a two-tower joint space suitable for flexible cross-modal applications.
- Audio representation learning: Audio representation learning has used supervised, unsupervised, and self-supervised pretraining for general environmental audio and music.AudioSet-pretrained AST models and discriminative or generative objectives provide important comparison points for transfer learning.
- Cross-modal supervision: MuLan develops a cross-modal supervision recipe from abundant text annotations weakly associated with music audio and benchmarks the resulting representations for transfer learning.The study also evaluates different audio encoder architectures.
- Cross-modal contrastive learning: Cross-modal contrastive learning has aligned image, text, audio, and tags, extending shared representation learning across modalities.Earlier audio work aligned latent audio representations with associated tags using contrastive learning.
- Music-text models: Music retrieval has commonly used large vocabularies of mined n-grams or other supervised objectives to ground free-form text in audio.These approaches include cross-entropy training with text-label embeddings and tasks connected through preexisting emotion labels.
- Music-text models: Unlike MuLaP’s early-fusion multimodal Transformer, MuLan’s two-tower architecture produces a joint embedding space for arbitrary music audio and natural-language queries.This design supports cross-modal retrieval, zero-shot tagging, and language understanding beyond transfer learning applications.
3. PROPOSED APPROACH
MuLan constructs a shared music-audio and free-form-text embedding space with separate encoders trained by cross-modal contrastive learning. The approach uses large-scale mined audio-text pairs and supports flexible semantic connections between arbitrary music audio and natural-language descriptions.
- Learning framework: MuLan uses a two-tower architecture whose separate audio and text networks terminate in same-dimensional, ℓ2-normalized embedding spaces.The audio network processes log mel spectrogram windows, while the text network processes padded or truncated token sequences.
- Training data: Training samples pair randomly selected spectrogram context windows with randomly selected associated text annotations from the same recording.Each mini-batch contains target audio-text pairs sampled from recordings and their associated annotations.
- Learning framework: MuLan minimizes a batch-wise cross-modal contrastive loss that increases compatibility for target audio-text pairs and decreases it for non-target pairs.The critic applies an exponentiated temperature-scaled inner product, which is effectively cosine similarity for normalized embeddings.
- Training dataset mining: The training corpus contains approximately 44 million filtered 30-second music clips, paired with short-form, long-form, playlist, and AudioSet text annotations.Text sources are noisy, so curated sentence classification and rule-based heuristics filter some short- and long-form annotations, while playlist text remains unfiltered.
4. EXPERIMENTS
The experiments train MuLan audio-text models on the mined music recordings and processed text sources, using two audio encoders and a shared BERT text encoder. Training uses large batches, Adam optimization, scheduled learning-rate decay, and a trainable contrastive temperature.
- Training setup: MuLan models are trained for 14 epochs on 44M recordings with AudioSet, short-form, long-form, and playlist text sources.Both M-Resnet-50 and M-AST use BERT-base-uncased as the text encoder.
- Optimization: M-Resnet-50 uses batch size B = 6144 pairs, whereas M-AST uses B = 5120 pairs because of memory limitations.The temperature is initialized to τ = 0.1 for both models.
4.1 Evaluation Tasks
The evaluation tests MuLan’s similarity-based zero-shot tagging, transfer learning, retrieval, and music-text language understanding. Benchmarks include MagnaTagATune, AudioSet, curated playlists, and text triplet tasks, with explicit caveats about label exposure.
- Zero-shot Music Tagging: Zero-shot tagging scores candidate labels by cosine similarity between the clip’s audio embedding and each tag string’s text embedding.A contextual text encoder and cross-modal contrastive learning provide a prediction space that can extend to potentially unseen labels.
- Zero-shot Music Tagging: MagnaTagATune evaluation uses both top-50 and full 188-tag settings, reporting class-balanced AUC-ROC on standard test partitions.AudioSet evaluation is also included in the tagging benchmarks.
- Zero-shot Music Tagging: AudioSet is included in contrastive training and overlaps with some MagnaTagATune classes, so these evaluations are not strictly zero-shot from a label-exposure perspective.The comparison estimates the cost of a flexible natural-language interface that also supports classes outside the AudioSet ontology.
- Transfer Learning with Linear Probes: For transfer learning, an independent per-class logistic-regression layer is trained on frozen 128-dimensional MuLan audio embeddings using the benchmark training sets.The evaluation follows prior transfer-learning protocols for direct comparison.
- Cross-modal Retrieval: MuLan retrieves music clips closest to a text query in the shared embedding space using a proprietary collection of 7,000 expert-curated playlists.The playlists have titles, descriptions, and 10–100 recordings, and do not overlap with playlist information used in training.
- Text Triplet Classification: Text understanding is evaluated with triplet classification, where a text embedding is judged correct when the positive string is closer than the negative string to the anchor.One evaluation set uses AudioSet labels, long-form descriptions, and sampled negative descriptions.
4.2 Results and Discussion
MuLan supports music tagging, text retrieval, and music-domain language understanding, with performance depending on text sources and semantic alignment. Its audio embeddings achieve state-of-the-art transfer learning, while zero-shot tagging remains vulnerable to ambiguous and negated labels.
- Zero-shot tagging: Zero-shot tagging is degraded when evaluation labels have ambiguous senses or simple negation that the text encoder does not adequately model.The text embeddings of phrases such as “not rock” can remain similar to those of “rock.”
- Text ablation: Including more text sources generally improves downstream performance, while the loss on AudioSet AUC remains relatively minor.AudioSet-only training performs best on AudioSet evaluation because the text encoder learns the test-label semantics directly.
- Music tagging: State-of-the-art transfer learning performance is achieved on all evaluated music tagging tasks using linear probes on MuLan audio embeddings.The results indicate that the pretrained audio encoder produces general-purpose music representations while supporting natural-language applications.
- Music retrieval from text queries: Large-scale short-form tags improve fine-grained query grounding, while comments and playlist data help with complex, compositional, and negated queries.Examples include “instrumental action movie soundtrack,” “classical music with middle eastern influence,” and “hard rock without vocals.”
- Text triplet classification: Long-form music text annotations produce a specialized text embedding model that outperforms generic sentence embedding models on triplet classification.This specialization is achieved through cross-modal training without a text-only fine-tuning loss.
5. CONCLUSIONS
The paper presents a music audio–natural-language joint embedding model trained on weakly paired data at unprecedented scale. MuLan provides a versatile natural-language interface and state-of-the-art transfer learning performance, while improved text filtering remains an avenue for handling rare and subtle language constructs.
- CONCLUSIONS: MuLan is a music audio–natural-language joint embedding model trained with an unprecedented scale of weakly paired text and audio data.The model is presented as a first attempt at a free-form natural-language interface for music audio.
- CONCLUSIONS: MuLan’s natural-language interface supports a range of applications, while its pretrained audio embeddings achieve state-of-the-art performance on music tagging benchmarks.The conclusion characterizes the interface as versatile across the demonstrated applications.
- CONCLUSIONS: Improved text filtering methods may better distinguish weak signal from absolute noise and improve handling of rare and subtle language constructs.The paper identifies this as an area with substantial room for improvement.