Source-linked AI summary
Multimodal Music Recommendation System using LLMs
Srikar Prabhas Kandagatla, Sreehitha R. Narayana, Chandana Magapu, Swetha Mohan, Shamanth Kuthpadi, Hongjie Chen, Ryan A. Rossi, Franck Dernoncourt, Nesreen Ahmed
TL;DR
Music recommenders often treat songs as opaque interaction IDs, leaving acoustic, lyrical, and semantic content underused. This paper constructs a multimodal LastFM-1K framework and benchmark that integrates embeddings, LLM-generated metadata, and completion ratios into E4SRec-based session recommendation. Multimodal enrichment generally improves recommendation quality, although gains vary across architectures and fusion configurations.
Problem
Existing ID-based and LLM-based recommenders do not consistently ground sequential music recommendation in jointly modeled acoustic, lyrical, semantic, and engagement content.
Method
The paper enriches LastFM-1K with audio and lyric embeddings, LLM-generated MGPHot metadata, and listening completion ratios, then evaluates them in E4SRec across sequential and LLM backbones.
Results
Multimodal enrichment generally improves recommendation quality, with the strongest gains observed for Qwen2.5-7B-Instruct combined with BERT4Rec.
Takeaways & Limitations
Semantic and acoustic information provide signals beyond interaction-only representations, but their benefits depend on the backbone and fusion configuration.
Takeaways & Limitations
Improvements are not consistent across all backbone and encoder combinations, suggesting that current multimodal fusion can introduce noisy or weakly aligned information.
Abstract
from arXiv · showhide
Music recommendation systems typically treat songs as opaque tokens, relying on collaborative interaction histories which overlooks semantic or acoustic content. Prior work has explored LLM-augmented, multimodal, and text-enhanced approaches to sequential recommendation, and while some methods partially combine semantic, acoustic, or engagement signals, none jointly model all three within a unified LLM-based sequential reasoning framework that grounds recommendations in actual song content. In this work, we propose a multimodal framework for session-based music recommendation that enriches the LastFM-1K dataset with three complementary signals: (1) audio and lyric embeddings extracted using pretrained music and text representation models, (2) LLM-generated semantic metadata using the MGPHot annotation schema, and (3) listening completion ratios. We adopt the E4SRec framework by extending it with multimodal features and different item ID encoder backbones, including SASRec, BERT4Rec, and GRU4Rec. We further extend the LLM backbone option with LLaMa-2-13B, Qwen2.5-7B-Instruct, and LLaMa-3-70B in both zero-shot and fine-tuned settings. Our experiments show that integrating content-based features improves over ID-only baselines up to 95% in terms of Recall and 79% in terms of NDCG. Moreover, our experiments show that naive multimodal fusion does not always yield additive improvements, highlighting challenges in cross-modal integration. We release a large-scale multimodal benchmark for music recommendation.
1 Introduction
Existing sequential music recommenders often represent songs only as interaction IDs, limiting their ability to use acoustic, lyrical, and semantic content. The paper addresses this gap with a multimodal, LLM-based framework and benchmark that combine content and engagement signals.
- ID-based recommenders treat tracks as independent tokens, limiting reasoning about acoustic similarity, lyrical themes, and semantic context under sparsity and cold-start conditions.
- LLM-based recommenders model sequential behavior but often rely on ID embeddings or weak textual proxies rather than actual song content.
- The framework enriches LastFM-1K with audio and lyric embeddings, LLM-generated semantic metadata, and listening completion-ratio signals.
- Experiments extend E4SRec across SASRec, BERT4Rec, and GRU4Rec backbones, several LLMs, four fusion strategies, and zero-shot and fine-tuned settings.
- The released benchmark supports future research on integrating multimodal features into LLM-based recommendation systems.
2 Related Work
Related work spans sequential, session-based, multimodal, and music-representation recommendation. These lines of research motivate combining behavioral, semantic, and acoustic representations for session-based music recommendation.
- Sequential Recommendation: Sequential recommendation models chronological behavior, with recent Transformer, graph, and LLM approaches targeting long-range dependencies and multimodal user behavior.
- Sequential Recommendation: E4SRec integrates pretrained item ID embeddings into an LLM-based sequential recommender, while PISA and IntervalLLM model music listening patterns and irregular timing.
- Session-based Recommendation: Session-based recommendation predicts the next item from short-session interactions, reflecting rapidly shifting intent in music streaming.
- Multimodal Recommendation: Multimodal recommendation combines heterogeneous content signals such as text, audio, images, and metadata with interaction data.
- Representation for Music Recommendation: Music representation learning has progressed from collaborative ID vectors toward rich, multimodal, and semantically structured embeddings, especially relevant under sparsity and cold-start conditions.
3 Multimodal Data Curation Pipeline
The paper builds a multimodal LastFM-1K benchmark by adding semantic, acoustic, lyrical, and behavioral signals to session-based listening data. Its curation pipeline combines pretrained embeddings, LLM-derived musicological features, and completion ratios for recommendation experiments.
- The pipeline enriches LastFM-1K with song-level semantic, acoustic, lyrical, and behavioral signals for zero-shot and fine-tuned sequential recommendation.
- The processed dataset contains 814 users, 4.2 million listening events, and 295,957 unique tracks after filtering and session segmentation.
- MGPHot provides 58 interpretable attributes grouped into Lyrics, Vocals, Harmony, Rhythm, Instrumentation, Sonority, and Composition.
- Extended LLM-extracted metadata targets meaning, structure, and cultural context beyond audio encoders, genre tags, and standard Spotify attributes.
- Audio features combine handcrafted acoustic descriptors with neural embeddings from CLAP, MERT, Music2Vec, EnCodec, and MFCC-based representations.
- Lyric embeddings use pretrained text encoders, while LLM prompts rate MGPHot attributes and extract extended features from track metadata and available lyrics.
- Completion ratios distinguish near-complete listening from early skips, providing an engagement signal that conventional interaction modeling treats identically.
4 Recommendation Method
The method represents songs with item IDs, audio, lyrics, semantic metadata, and engagement signals, then uses a session encoder and multimodal fusion to rank the next song. It compares fusion strategies that differ in simplicity, parameter efficiency, and cross-modal expressiveness.
- The framework predicts the next song by ranking candidate items from a user's listening session.
- Multimodal Item Representations: Each song combines item ID, audio, lyric, and metadata embeddings through a multimodal fusion function.
- Session Encoding: The session encoder maps engagement-aware item representations to a user-state vector for next-item prediction.
- Content-Aware Representation: Audio, lyric, and semantic features enrich ID-based representations with content-level information for comparing songs beyond opaque identifiers.
- Engagement-Aware Signaling: Completion ratio distinguishes stronger listening behavior from incidental or incomplete exposure within a session.
- Multimodal Fusion: Four fusion strategies compare direct concatenation, weighted combination, cross-attention, and modality-based modulation of item representations.The strategies trade off simplicity, parameter efficiency, and cross-modal expressiveness.
5 Experiments
Experiments show that multimodal music recommendation can improve over ID-only modeling, but gains depend strongly on the LLM backbone, item encoder, fusion strategy, and feature combination.
- Overall findings: Multimodal augmentation often improves Recall@K and NDCG@K over interaction-only models, indicating that content signals complement ItemID sequences.The improvement is not uniform across configurations.
- LLM and encoder effects: 0.121 Recall@20 is achieved by BERT4Rec with Qwen2.5-7B-Instruct after multimodal augmentation, improving from 0.102.The same setting illustrates that benefits can depend on the sequential encoder.
- Representation choices: MPNet provides the strongest overall ranking quality among lyric embedders, achieving the best NDCG and MRR across cutoffs.BERT and BGE-M3 perform near zero at tight cutoffs, while MiniLM favors Recall and BGE-M3 favors broader retrieval.
- Metadata ablation: Naive metadata fusion can reduce performance: concatenating Spotify and MGPHot sources lowers NDCG@10 by 38% relative to the E4SRec baseline.The baseline reaches 0.026, while the combined 76-column prompt reaches 0.016.
- Fusion and backbone effects: LLaMa-3-70B is not uniformly stronger: its best gains depend on the item backbone and fusion strategy, with Audio FiLM raising SASRec NDCG@20 from 0.036 to 0.045.Other configurations show different preferences, including completion-ratio signals for GRU4Rec and concatenation for BERT4Rec.
6 Conclusion
The paper presents a multimodal LastFM-1K benchmark and evaluates semantic, acoustic, lyrical, and behavioral signals across recommendation architectures and LLM backbones. Results generally favor multimodal enrichment, but improvements vary across backbone and encoder combinations, motivating more robust fusion and alignment.
- The benchmark enriches LastFM-1K with semantic metadata, audio representations, lyric embeddings, and engagement signals for multimodal recommendation evaluation.
- Multimodal features are evaluated across sequential recommendation architectures and LLM backbones in zero-shot and fine-tuned settings.
- Observed improvements are not consistent across backbone and encoder combinations, suggesting noisy or weakly aligned information from current fusion approaches.
A Additional Implementation Details
The implementation combines standard machine-learning libraries with pretrained audio and text encoders, multiple LLM backbones, and external music-data retrieval services. LLM-generated MGPHot annotations use a three-model consensus, while larger models may require quantization.
- The framework uses PyTorch, HuggingFace Transformers, PEFT, NumPy, Pandas, and Librosa for training, fine-tuning, preprocessing, and acoustic feature computation.
- Pretrained audio encoders include CLAP, MERT, Music2Vec, and EnCodec, while lyric representations use MiniLM, BGE-M3, MPNet, MultiLG, and BERT.
- Evaluated recommendation backbones include LLaMa-2-7B, LLaMa-2-13B, LLaMa-3-70B, and Qwen2.5-7B-Instruct in zero-shot and LoRA fine-tuned settings.
- MGPHot perceptual annotations are obtained through consensus among LLaMa-3.3-70B-Instruct, Qwen2.5-7B-Instruct, and Mistral-Nemo-12B-Instruct.
- Audio retrieval uses yt-dlp, while structured attributes such as valence, energy, and danceability come from ReccoBeats and SpotifyEA APIs.
B MGPHot Annotation Validation
The annotation validation compares generated MGPHot ratings with published ground truth on their overlapping song set. Across categories, generated annotations preserve positive rank-order agreement while maintaining relatively low absolute error.
- Validation compares generated MGPHot annotations with published ground-truth ratings over the intersection of the top-50k LastFM-1K songs and the MGPHot release.
- For all 58 MGPHot attributes, validation computes Spearman’s ρ and mean absolute error on the normalized [0, 1] scale, aggregated by category.
- Across every category, mean Spearman’s ρ is positive and mean MAE remains below 0.25, indicating preserved rank ordering within a tight error band.
- Agreement is strongest for Lyrics (ρ≈0.56) and Sonority (ρ≈0.53), followed by Instrumentation (ρ≈0.50).