Source-linked AI summary
Content-driven Music Recommendation: Evolution, State of the Art, and Challenges
Yashar Deldjoo, Markus Schedl, Peter Knees
arXiv:2107.11803v2cs.IRcs.MM
TL;DR
Large online music collections make discovery difficult, and the survey addresses the need to understand how content data is categorized and used in music recommendation. It reviews 55 articles through a five-layer content model and six challenge areas, synthesizing the surveyed approaches and the continuing properties and limits of content sources.
Problem
Expanding online music collections make it difficult for users to discover engaging music or identify a specific piece among ambiguities and versions.
Source: §1 · p. 1Method
The survey reviews 55 articles using a five-layer onion model of music content and organizes its discussion by six recommendation challenges.
Source: §6 · pp. 23–24Results
The synthesis identifies six overarching challenges, from diversity and explanations to context, sequences, scalability, and cold start.
Source: §6 · p. 24Takeaways & Limitations
The model characterizes content sources from signal to derivative data, while the survey maps reviewed work by content layer, goals, and methods.
Source: §6 · pp. 23–24Subjective content makes it more challenging to derive general item representations without introducing the user dimension.
Source: §3 · p. 6
Abstract
from arXiv · showhide
The music domain is among the most important ones for adopting recommender systems technology. In contrast to most other recommendation domains, which predominantly rely on collaborative filtering (CF) techniques, music recommenders have traditionally embraced content-based (CB) approaches. In the past years, music recommendation models that leverage collaborative and content data -- which we refer to as content-driven models -- have been replacing pure CF or CB models. In this survey, we review 55 articles on content-driven music recommendation. Based on a thorough literature analysis, we first propose an onion model comprising five layers, each of which corresponds to a category of music content we identified: signal, embedded metadata, expert-generated content, user-generated content, and derivative content. We provide a detailed characterization of each category along several dimensions. Second, we identify six overarching challenges, according to which we organize our main discussion: increasing recommendation diversity and novelty, providing transparency and explanations, accomplishing context-awareness, recommending sequences of music, improving scalability and efficiency, and alleviating cold start. Each article addresses one or more of these challenges is categorized according to the content layers of our onion model, the article's goal(s), and main methodological choices. Furthermore, articles are discussed in temporal order to shed light on the evolution of content-driven music recommendation strategies. Finally, we provide our personal selection of the persisting grand challenges which are still waiting to be solved in future research endeavors.
1. Introduction and Motivation
The growth of online music collections makes discovery difficult, while approaches based only on metadata can yield predictable recommendations. The survey frames content-driven music recommendation as a way to use item content alongside collaborative signals and organizes prior work around a content model and key challenges.
1. Introduction and Motivation
- Expanding streaming collections make it harder to discover engaging music and even to identify a specific piece among ambiguities or versions.The introduction describes this as searching for a needle in a haystack.Source: §1 · p. 1
- Pure collaborative filtering disregards item-content knowledge, whereas content-driven models combine collaborative and content information.The survey links content-driven recommendation with explanation, musical-taste exploration, and cold-start use cases.Source: §1 · p. 2
- Metadata-only recommendations can be predictable, while expressive audio and multimedia features can support more informed, less trivial recommendations.An artist-based system may recommend songs the user would have listened to anyway.Source: §1 · p. 2
- The survey proposes a hierarchical model of music-item characteristics, categorizes state-of-the-art approaches, and outlines persisting grand challenges.It asks how content features can be categorized, which techniques use them, and how content can help address recommendation challenges.Source: §1 · p. 2
2. Survey Context and Literature Search Methodology
The survey distinguishes its content-focused music recommendation scope from earlier surveys centered on context, sequences, deep learning, or music information retrieval. It uses a multi-stage search across venues and scholarly indexes, while noting that its publication list is not exhaustive.
2.1. Related Surveys
- Unlike earlier surveys focused on context, general sequence recommendation, or content-based music retrieval, this survey centers content data in music recommendation and its role in recommendation challenges.It also extends earlier automatic-playlist-generation surveys by reviewing how content features enter sequential recommendation.Source: §2.1 · p. 3
2.2. Selection of Relevant Publications
- The search combined targeted venue and keyword searches with journal and scholarly-index searches, followed by citation and related-work checks.It covered relevant papers from RecSys and ISMIR histories and papers published from 2010 to 2020 in other selected venues.Source: §2.2 · p. 4
- The search yielded 47 publications, but the authors explicitly do not claim the resulting list is exhaustive.They describe it as an overall picture of recent advances, trends, and challenges.Source: §2.2 · p. 4
3. Levels of Content
The onion model organizes music content from the audio signal outward through increasingly semantic and subjective sources. These layers differ in timing, context, community dependence, cold-start exposure, data diversity, noise, and credibility.
3. Levels of Content
- The model treats content broadly and arranges five layers from audio through embedded metadata, expert-generated and user-generated content, to derivative content.Moving outward adds semantic valence and subjectivity as data relates more to music practices and usage.Source: §3 · pp. 5–6
- Embedded metadata describes and identifies a music item, and can link audio to expert-, user-generated, and derivative sources.Examples include artist, track, and album information, lyrics, recording year, and associated multimedia.Source: §3 · p. 5
- User-generated content is analyzed to recommend associated music, while derivative content covers reuse or re-contextualization such as remixes, covers, and music placed in films or ads.The survey notes derivative content has received less attention in content-based music recommendation and may be a future direction.Source: §3 · p. 6
- Moving outward shifts from numeric signal descriptors toward semantically charged, culturally produced data and more subjective categories.Subjectivity makes general item representations harder to derive without introducing a user dimension, although large-scale subjective data may provide a better recommendation basis.Source: §3 · p. 6
- Outer layers depend more on community contributions and face greater item cold-start risk, data scarcity, noise, and credibility concerns than audio analysis.A larger volume of data, ideally from independent sources, is needed for outer-layer sources to achieve comparable trust and credibility.Source: §3 · pp. 6–7
4. Music Recommendation Exploiting Onion-Model Content as Side Information
The section organizes content-driven music recommendation by the onion-model content it uses and reviews classical, neural, graph-based, and two-phase strategies for integrating that content as side information.
4. Music Recommendation Exploiting Onion-Model Content as Side Information
- The survey groups approaches by the onion-model content they exploit and describes each technique class before giving music-recommendation examples.It covers classical model-based CF, deep neural models, graph-based approaches, and a two-phase approach.Source: §4 · p. 7
4.1. Classical model-based CF
- Classical methods extend memory-based CF with content similarity or extend model-based CF, while audio features can create scaling complications for factorization machines.Textual side data is typical; high-dimensional audio features may complicate FMs, especially when higher-order interactions are included. Pure CBF using audio underperforms CF, while outer-layer user-generated tags can support cold-start recommendation.Source: §4.1 · p. 7
4.2. Deep neural models
- Deep neural models can represent audio as latent features and combine content and interaction representations through fusion, aggregation, or concatenation.Multi-phase learning uses fewer resources and is easier to scale, whereas end-to-end learning produces task-specific representations and is more accurate.Source: §4.2 · pp. 7–8
4.3. Graph-based approaches
- Content-aware graph methods combine graph structure with node content, using heterogeneous graphs, modality-specific preferences, or attention to learn recommendation representations.Examples include GASM and HIGE for music recommendation, and multimodal systems such as MMGCN, GRCN, HHFAN, and MGAT.Source: §4.3 · p. 8
4.4. Other approaches
- For audio recommendation at scale, a two-phase approach uses CF to narrow millions of items to hundreds, then audio-based CBF reranks them to select tens.The initial candidate-generation stage may be prefetched on mobile devices based on previous interactions.Source: §4.4 · p. 8
5. Main Challenges
This section surveys content-driven music recommendation research organized around six challenges, tracing how methods use different content layers. It covers diversity and novelty, transparency and explanations, context awareness, sequencing, efficiency, and cold start.
5. Main Challenges
- The review identifies six challenges and organizes articles by goals, methods, and the onion-model content layers they use.Across the reviewed papers, audio appears in 68.3%, embedded metadata in 32.7%, expert-generated content in 40.0%, user-generated content in 52.7%, and derivative content in 1.8%; audio-level features have become more common in recent years alongside complex DNNs and graph-learning methods.Source: §5 · p. 8
5.1. Increasing recommendation diversity and novelty
- Diversity and novelty can counter filter bubbles and popular-item bias, while balancing discovery against recommendation accuracy.Collaborative filtering can reinforce users’ existing tastes and favor popular items; content-based or hybrid approaches using audio similarity can increase novelty. Diversity also matters in playlists, where it can be pursued alongside coherence.Source: §5.1 · p. 9
- Hybrid and graph-based approaches, user-group tailoring, and metadata-aware optimization offer ways to increase recommendation diversity.The hybrid model improved accuracy and artist variety; the system increased diversity while improving similarity estimation. Combining CF, CBF, and popularity worked best for recall and F-measure among users with high diversity preference and medium or high novelty preference [76]; a metadata-aware approach recommended approximately 2.5 times more artists from different countries than its best baseline. A deep-and-wide model [78] performed well on diversity and novelty while remaining competitive on precision and NDCG; users’ music sophistication and visual memory shaped perceived diversity in a study of 83 participants.Source: §5.1 · pp. 9–11
- Novelty methods include browsing-graph transformation [80], while hubness mitigation targets distortions that make some audio-similar tracks recur in recommendation lists.The browsing graph offered more novel items while maintaining accuracy [80]. Sequential recommendation work combines freshness with other playlist goals, and a memory-based model using exposure frequency and recentness outperformed several baselines. Hubness methods include smoothing audio-rating models and mutual proximity normalization.Source: §5.1 · pp. 11–12
5.2. Providing transparency and explanations
- Transparency reveals how a recommender works, whereas explainability tells users why items were recommended; the review groups explanations into five types.The categories are neighborhood-based, feature-based, contextual, audible, and user-characteristics-based explanations. Prior studies have found that transparency and explainability can increase engagement and trust.Source: §5.2 · p. 12
- Neighborhood-based explanations draw on similar users or items, while feature-based explanations use content descriptors to explain recommendations.Examples include explaining a recommendation through similar users’ ratings, or through audio, genre, or cover-version features. MusicSun uses descriptive terms both to steer recommendations and explain them, while artist tag clouds can show shared and distinguishing descriptors [97, 98].Source: §5.2 · pp. 12–13
- Contextual explanations use factors such as mood or listening time, while audible explanations provide spoken or music-based rationales.MoodPlay combines mood and audio similarity and lets users explore music through mood terms; BART personalizes explanations to context, with offline and A/B tests supporting improved satisfaction. Audible explanations can use radio-style voice or relevant audio segments.Source: §5.2 · p. 13
- Explanation approaches include playlist-transition voice narratives, listenable audio explanations, and models that use users’ positive or negative preferences [101],,.LEMONS uses AudioLIME to weight time–source audio components; CLEP uses Siamese networks and audio features, with CLEP-N achieving the highest accuracy and precision and lowest false-positive rate among its three variants.Source: §5.2 · p. 14
- User studies find that explanation preferences vary by personal characteristics: low-sophistication users prefer simple explanations, while low-openness users prefer fewer styles, especially bar charts [104].Both high- and low-need-for-cognition users prefer receiving all explanations at once.Source: §5.2 · p. 14
5.3. Accomplishing context-awareness
- Spatial recommenders link music to places using shared tags and knowledge graphs, latent song–venue representations, or GPS-derived user distances [114],,.The POI system combines auto-tagging and knowledge-graph recommendations through Borda rank aggregation; the venue model represents songs and venues in a shared latent topic space and outperforms audio-only approaches on precision and NDCG for most venue types. Hybrid systems using location information show higher accuracy than single-source approaches.Source: §5.3 · pp. 15–16
- GPS-based running recommendations infer emotion from route attributes, while workplace recommendations use audio characteristics to account for concentration; both evaluations have important methodological or generalizability limitations [117],.The running app was evaluated on service response times rather than recommendation relevance or utility; the workplace study had relatively few participants and did not directly validate concentration effects before evaluating observable behavior.Source: §5.3 · p. 16
- Affective recommenders infer mood from audio or user-generated text, and combine mood labels or features with listening data to personalize recommendations, [121],.Mood Flows uses expert annotations and audio-based classifiers to create large-scale personalized playlists; Chen et al. report best MAP and recall when metadata, audio, text, and mood features are combined, while Deng et al. find user-based CF outperforms their other variants on precision, recall, and F1. The LiveJournal-based results may not generalize to other data sources.Source: §5.3 · pp. 17–18
- Social-context methods improved recommendation outcomes: among 68 app users, 47% of socially recommended songs received at least three stars versus 33% for non-social recommendations, and influence-weighted CF lowered normalized RMSE versus standard user-based CF.Source: §5.3 · p. 18
- Cultural context improves music recommendation: combines preference, cultural, and socioeconomic features, while finds four country/region variants outperform a context-free VAE and standard baselines.The user model represents music preference as the mean of listened-track Spotify audio features; the four variants use country or music-cultural-region information in a VAE, with no significant differences among the variants.Source: §5.3 · p. 19
5.4. Recommending sequences of music
- Content-driven automatic playlist generation uses semantic features to create coherent sequences, spanning location-aware playlists, Markov-chain embeddings, and two-stage next-track reranking.GeoShuffle incorporates location, path, time, and listening history; logistic Markov embeddings model track transitions and can generate or continue playlists; the two-stage method selects candidates from recent listening history, then reranks them for desired continuity.Source: §5.4 · p. 19
- Song-tagged attentive neural networks outperformed several baselines, while a dynamically updated recommender required less interaction data and online learning-to-rank showed significant gains in early-stage performance and long-term convergence.Source: §5.4 · p. 20
5.5. Improving scalability and efficiency
- Scalability work accelerates retrieval through audio-signature hashing, artist-similarity search pruning, and incrementally updated hybrid audio-CF models.The hashing system was evaluated on collections from 1K to 114K pieces; artist-based pruning drastically reduced audio-similarity computations, with the reduction controlled by SOM size. The incremental model took 10 minutes to train initially and 5 seconds to update, but used a small, Japan-only collection.Source: §5.5 · pp. 21–22
- Music-video prefetching combines content-based and collaborative recommenders, with user-based CF achieving the strongest F1 while user-independent content methods offer greater privacy.The pipeline combines personalized and user-independent content-based prefetching with ALS and memory-based CF; evaluation used about 14K videos watched by 5K users, and the authors note that the content-based methods were rather simple.Source: §5.5 · p. 21
5.6. Alleviating cold start
- Cold-start methods exploit content available before interactions, from raw audio for new tracks to playlist titles and user-conditioned audio embeddings.The onion model links greater cold-start risk to content farther from the signal layer; ’s Siamese audio-similarity model achieved 81.64% accuracy on 6,400 music pairs, and 55% of recommended songs had fewer than 1,500 plays. retrieves tracks from title-matched playlist clusters on the million-playlist MPD, while matches new-track audio embeddings to user embeddings.Source: §5.6 · pp. 22–23
- [171] estimates attributes for cold-item rating prediction, while jointly addresses new users and items and multimodal PMF [173] switches modalities when interactions are absent.[177] uses user-item-tag tensors to address sparse social-tag data, and combining tags with acoustic features improves recommendation quality over either alone; a noted disadvantage of related tag studies is their frequent omission of user-item interactions.Source: §5.6 · p. 23
6. Conclusions and Open Grand Challenges
The survey organizes 55 articles with a five-layer model of music content and six overarching recommendation challenges. It highlights persistent open questions and presents the survey as a reference for future work.
6. Conclusions and Open Grand Challenges
- The onion model arranges music content from signal through embedded metadata, expert-generated and user-generated content to derivative content, characterizing each layer across several dimensions.The dimensions include temporal component, semantics, subjectivity, cultural context, community, cold start, data diversity, quality, and credibility.Source: §6 · p. 23
- The authors present the onion model as a meaningful way to categorize musical data and highlight trade-offs, and hope the survey serves as a standard and reference for further research.They describe the review as a reference to the evolution and state of the art in content-driven music recommendation.Source: §6 · p. 24
- The survey organizes its discussion around six challenges: diversity and novelty, explanations, context-awareness, music sequences, scalability and efficiency, and cold start.It also categorizes reviewed work by content layer, goals, and methodological choices.Source: §6 · p. 24
- Derivative content, including remixes, covers, and parodies, has been largely neglected in music recommendation, despite possible relevance to culture-aware systems.Acquiring, processing, understanding, and integrating it remain open challenges.Source: §6 · p. 24
- Open questions include how cognitive models of decision-making among many choices should inform recommenders, with music content potentially supporting psychology-informed models through memory-linked categories.The passage gives acoustic characteristics and genre as examples of content categories tied to human memory.Source: §6 · p. 24
- Diversity and novelty require multifaceted evaluation, but the importance of existing metrics for real-world recommenders and their relationship to user satisfaction remain unclear.The metrics span content and interaction as well as intra-list and inter-list perspectives.Source: §6 · p. 24
- Fairness research must consider consumers and content creators or providers, including gender opportunity, and investigate trade-offs among user fairness, item fairness, and accuracy.The survey identifies balancing these factors as an open research direction.Source: §6 · p. 24
- For conversational music recommenders, leveraging content levels, choosing interaction modes, and presenting recommendations remain open design questions.The passage contrasts natural language with spoken audio and unimodal with multimodal presentation.Source: §6 · p. 24