Source-linked AI summary
JamendoMaxCaps: A Large Scale Music-caption Dataset with Imputed Metadata
Abhinaba Roy, Renhang Liu, Tongyu Lu, Dorien Herremans
TL;DR
Music-language research lacks large-scale datasets with rich musical and textual annotations. JamendoMaxCaps addresses this gap with over 362,000 instrumental tracks, generated captions, retrieval-assisted metadata imputation, and a local language model, yielding high subjective and objective evaluation scores.
Problem
Large-scale, high-quality datasets with rich musical and textual annotations remain limited, constraining music-language understanding research.
Method
JamendoMaxCaps combines generated captions, multimodal retrieval of similar songs, and local-language-model in-context learning to impute missing metadata.
Results
All listening-study metrics scored above 5.09 out of 7, with speed matching at 5.82 and vartags matching at 5.59.
Takeaways & Limitations
The publicly available dataset provides a comprehensive resource for music retrieval, multimodal representation learning, and generative music research.
Takeaways & Limitations
Underrepresented genres may receive less reliable imputations because scarce relevant context can reinforce dominant-category bias.
Abstract
from arXiv · showhide
We introduce JamendoMaxCaps, a large-scale music-caption dataset featuring over 362,000 freely licensed instrumental tracks from the renowned Jamendo platform. The dataset includes captions generated by a state-of-the-art captioning model, enhanced with imputed metadata. We also introduce a retrieval system that leverages both musical features and metadata to identify similar songs, which are then used to fill in missing metadata using a local large language model (LLLM). This approach allows us to provide a more comprehensive and informative dataset for researchers working on music-language understanding tasks. We validate this approach quantitatively with five different measurements. By making the JamendoMaxCaps dataset publicly available, we provide a high-quality resource to advance research in music-language understanding tasks such as music retrieval, multimodal representation learning, and generative music models.
I. INTRODUCTION
JamendoMaxCaps addresses the shortage of large-scale, richly annotated music-language data with over 362,000 freely licensed instrumental tracks, generated captions, and imputed metadata. Its retrieval-assisted local-language-model pipeline uses similar songs to infer missing fields.
- The work responds to limited large-scale, high-quality musical and textual annotations, which constrain progress in music-language understanding tasks.
- Over 362,000 freely licensed instrumental tracks form JamendoMaxCaps, expanding publicly accessible resources for music-language research.
- The dataset combines captions generated by a state-of-the-art music captioning model with imputed genre, tempo, mood, and instrumentation metadata.
- A retrieval system combines MERT musical features with sentence-transformer metadata representations to find similar songs for contextual metadata completion.
- A local large language model uses retrieved examples through in-context learning to populate incomplete metadata fields.
II. RELATED WORK
Prior work provides music-caption datasets, metadata-imputation methods, and pretrained retrieval representations, but substantial gaps remain in scale, audio coverage, and complete annotations. JamendoMaxCaps combines retrieval-based contextualization with local-model in-context learning to address these gaps.
- MusicCaps offers high-quality human captions but only around 5,000 examples, while MusicBench expands it to roughly 52,000 10-second instances.
- Midicaps exceeds 178,000 files but provides captions for MIDI music rather than audio recordings.
- Existing captioning approaches use symbolic features, tags, or pre-existing metadata, which may remain incomplete.
- Prior metadata-imputation studies infer missing attributes from musical embeddings or audio spectrograms using collaborative filtering and transformer models.
- CLAP and MERT demonstrate the utility of pretrained embeddings for music-text alignment, retrieval, and music representation learning.
- JamendoMaxCaps integrates musical representation, retrieval-based contextualization, and locally hosted large language models for scalable metadata imputation.
III. METHOD
The method generates captions from segmented audio and combines efficient inference with retrieval-based metadata imputation. Captioning produces multiple temporal descriptions at scale, while retrieval and prompting support subsequent metadata completion.
- A. Caption Generation: Captions are generated for consecutive 30-second clips, discarding final segments shorter than 15 seconds because they tend to produce lower-quality captions.
- A. Caption Generation: Only the first 30-second segment is used for subsequent retrieval and metadata-imputation tasks, despite retaining multiple captions as richer training data.
- A. Caption Generation: Batched inference and 4-bit quantization reduce memory demands, while captions are capped at 128 tokens and roughly 90% fit within that limit.
- A. Caption Generation: The standardized prompt targets instrumentation, genre, mood, rhythm, and listening scenarios, with explicit instrumentation and mood improving caption coherence.
- A. Caption Generation: The pipeline produces captions for more than 3 million 30-second audio segments, supporting later retrieval and metadata-imputation systems.
B. Retrieval System
The retrieval system represents songs with both musical-content and partially available metadata features, aligns their dimensions, and ranks songs by combined cosine similarity.
- Feature representation: MERT features represent musical content, while a sentence transformer encodes partially available metadata into a 768-dimensional vector.The MERT representation averages information across layers; the metadata embedding captures existing fields.
- Feature fusion: Sparse random projection aligns the MERT features with the 768-dimensional metadata embeddings before concatenation.The combined representation balances audio and metadata contributions using λ1 and λ2, constrained by λ1 + λ2 = 1.
- Similarity retrieval: Cosine similarity between concatenated query and dataset representations identifies songs similar in both musical and metadata features.For each song, the system retrieves the top 10 similar songs for downstream metadata imputation.
C. Metadata Imputation
Metadata imputation uses a locally hosted Llama-2 model prompted with captions and metadata from the top 10 songs retrieved for each target track.
- Language-model imputation: A locally hosted 7-billion-parameter Llama-2 model infers missing metadata while avoiding dependence on remote servers.The local deployment provides control over the imputation system.
- In-context construction: For each JamendoMaxCaps song, the pipeline retrieves the top 10 similar songs and concatenates their metadata and captions into in-context examples.These examples provide contextual information for the language model's prediction.
- Inference pipeline: The language model receives the in-context examples and target-song caption, then infers the target's missing metadata fields.The procedure is organized as retrieval, example construction, prompting, and metadata inference.
IV. EVALUATION AND STATISTICS
The section covers the dataset creation process before detailing the results of metadata imputation.
- Dataset creation: The section first discusses how the dataset is created.
- Metadata imputation: It then details the results of metadata imputation.
- Section organization: Dataset creation precedes the discussion of metadata-imputation results.
A. JamendoMaxCaps Dataset
JamendoMaxCaps combines Jamendo instrumental tracks, generated captions, and imputed metadata into a large music-caption dataset. The imputation broadens metadata coverage across genres, speeds, and variable tags.
- The dataset contains 362,238 Jamendo instrumental tracks after removing very short audio, with an average duration of 4 minutes and 12 seconds.
- Captions are generated with Qwen2-Audio from the first 30 seconds of each track for computational efficiency.
- Metadata imputation targets missing fields for 261,935 partially annotated songs, while 100,303 tracks already have complete metadata.
- Imputation adds underrepresented genres such as “Techno” and “Jazz” while retaining major categories including “Electronic” and “Rock”.
- Imputed speed metadata introduces categories such as “Fast”, “Slow”, “Very Fast”, and “Moderate”, while variable tags expand beyond “peaceful” and “energetic” to descriptors such as “relaxing,” “futuristic,” and “dreamy”.
- The resulting repository combines data collection, caption generation, and metadata imputation to provide richer descriptive information.
B. Retrieval System Settings
The retrieval system evaluates combinations of musical and metadata features, with λ settings controlling their relative contributions. A small subjective evaluation found the λ1 = 0.6 and λ2 = 0.4 configuration strongest, while broader evaluation remains a stated need.
- Qwen2-Audio caption quality was not evaluated because the authors treated the model as established, focusing instead on retrieval and metadata imputation.
- The retrieval evaluation compares metadata-only, musical-feature-only, and two mixed configurations by varying λ1 and λ2.
- λ1 = 0.6 and λ2 = 0.4 outperformed the other three retrieval configurations in a subjective evaluation.
- The evaluation used five songs, two independent evaluators, and a 7-point Likert scale to rate retrieved-song similarity.
- The authors identify evaluation with more participants as a future need for assessing retrieval configurations.
C. Objective Evaluation
The objective evaluation compares retrieval-based metadata imputation with generic in-context examples and assesses agreement with original metadata. It also frames semantic similarity as the primary evaluation target.
- C. Objective Evaluation: BERT-Score measures semantic similarity between imputed and original metadata for the genres, speed, and vartags fields.The evaluation generates imputed values for 5,000 examples per field with valid original metadata.
- C. Objective Evaluation: Retrieval examples consistently outperform generic examples across all reported BERT-Score and BLEU metrics for metadata imputation.The comparison uses 10 random examples as the generic baseline and evaluates 5,000 examples per metadata field.
- C. Objective Evaluation: Retrieval-based imputation reaches BERT-Scores of 0.58 for genres and 0.73 for speed.These values are reported as examples of the relatively high semantic-alignment scores obtained with retrieval examples.
- C. Objective Evaluation: The evaluation uses BLEU1 through BLEU4 alongside BERT-Score to measure n-gram overlap as well as semantic similarity.The same 5,000 examples per metadata field are used for this comparison.
D. Subjective Evaluation
The subjective evaluation asks listeners to judge the quality of imputed metadata across human-likeness and field-specific accuracy. Ratings are collected on a 7-point Likert scale from 15 participants.
- D. Subjective Evaluation: Fifteen participants take part in the listening study conducted through the PsyToolkit platform.Each participant listens to 10 randomly selected audio files and rates their associated imputed metadata.
- D. Subjective Evaluation: Participants rate whether metadata seems human-generated and assess genre, speed, and vartags accuracy.The study uses four criteria covering human-likeness and three metadata fields.
V. DISCUSSION
The discussion presents JamendoMaxCaps as a large-scale resource while identifying limitations from retrieved-example quality, retrieval bias, musical taxonomy subjectivity, and local-LLM capacity. These constraints motivate genre-aware retrieval and hybrid local-cloud refinement.
- V. DISCUSSION: Imputed metadata remains constrained by the quality and representativeness of retrieved examples, especially for genre and variable-tag descriptions.The discussion links lower BLEU scores for these fields to the difficulty of producing consistent labels.
- V. DISCUSSION: Musical taxonomy is subjective, so songs with unique stylistic elements may not align well with retrieved examples and can retain inconsistent labels.The same musical piece may receive different classifications depending on context and personal interpretation.
- V. DISCUSSION: Retrieval bias can make imputations more reliable for overrepresented genres such as Electronic and Rock than for underrepresented genres such as Jazz or Experimental.The authors warn that this imbalance may reinforce dominant categories in the imputed metadata.
- V. DISCUSSION: A locally hosted LLM offers cost, privacy, and control advantages but may lack the broader pretraining and fine-tuning of larger cloud-based models.The paper proposes hybrid local inference with selective cloud refinement as a future direction.
- VI. CONCLUSION: JamendoMaxCaps provides a large-scale, freely accessible dataset for music information retrieval, representation learning, and music generation.The dataset is positioned as a resource for advancing music-language understanding while addressing metadata sparsity.