Source-linked AI summary
Multi-Modal Semantic Expansion with Constrained LLM Reranking for Conversational Music Recommendation
Naman Garg, Sarika Jain, George Fazekas
TL;DR
The paper addresses conversational music recommendation under multimodal retrieval and multi-objective evaluation. It proposes a three-stage system combining multimodal retrieval, lightweight reranking, and persona-based generation, achieving a Blind B composite score of 0.3213 while finding that LLM injection is highly sensitive to scope. The reported Blind A injection results are observations across different evaluation settings rather than controlled comparisons.
Problem
Conversational music recommendation must combine evolving dialogue understanding, multimodal retrieval, response generation, and multiple competing evaluation metrics.
Method
The submitted system fuses seven dense embedding spaces, BM25, and artist matching, then applies lightweight reranking and persona-diversified GPT-4o-mini generation.
Results
The submitted system achieved a Blind B composite score of 0.3213, while differential-evolution RRF tuning improved MRR by +19.5% on the development set.
Takeaways & Limitations
LLM-guided artist injection showed a narrow operating range on Blind A, with conservative intervention outperforming broader injection in the reported observations.
Takeaways & Limitations
Blind A and Blind B use different session distributions, so their comparison is an approximate estimate rather than a controlled ablation.
Abstract
from arXiv · showhide
We present Team Semiintelligencn's solution for the ACM RecSys 2026 TalkPlayData Challenge, addressing conversational music recommendation through a multi-modal and personalized conversational recommender system. Our submitted system employs a three-stage pipeline: (1) multi-modal retrieval constructing decay-weighted centroids across seven dense embedding spaces - track- and user-level CF-BPR, Qwen3 (metadata, lyrics, attributes), CLAP audio, and SigLIP visual - supplemented by BM25 lexical retrieval and an artist substring-match signal, all fused via weighted Reciprocal Rank Fusion (RRF) with optimized signal weights; (2) lightweight reranking (history filtering, popularity smoothing, and catalog diversity penalization); and (3) persona-diversified response generation using GPT-4o-mini. Beyond this submitted configuration, we report development-time experiments with additional components - constrained LLM-guided artist injection, album continuation signals, XGBoost LambdaMART, and a superior GPT-4.1 response prompt - that were not deployed to Blind B due to cost and complexity constraints. We optimize RRF weights on a 500-session development split via differential evolution, improving MRR by +19.5%. On Blind A, we observe that unconstrained LLM-guided injection across 54 sessions causes catastrophic nDCG regression (-18.9%), while conservative injection on only 9 sessions yields the best observed Blind A nDCG - a finding we present as a Blind A observation warranting further validation. The submitted system achieves a Blind B composite score of 0.3213.
1 Introduction
The challenge requires conversational systems to retrieve personalized music and generate coherent responses from evolving dialogue. The paper addresses this through multimodal retrieval, constrained LLM use, and joint metric-aware design.
- Conversational recommendation must interpret evolving preferences, retrieve catalog items, and generate aligned textual responses across multiple turns.
- The TalkPlayData Challenge evaluates 20 ranked tracks and a natural-language response using retrieval, response-quality, lexical-diversity, and catalog-diversity metrics.
- The 47,071-track setting requires combining textual, acoustic, visual, and collaborative signals for named-entity and abstract mood queries.
- LLM-guided artist injection can degrade sharply beyond a narrow operating range, while independently optimizing one composite-metric component can harm others.
- The system contributes optimized multimodal retrieval, a conservative Blind A injection observation, and persona-diversified generation with 0.821 Distinct-2 across 80 sessions.
2 Task Formulation
The task uses a large multimodal music catalog and conversational sessions, with separate development, participant-visible, and held-out blind evaluations. Performance combines retrieval, response quality, lexical diversity, and catalog diversity.
- 2.1 Dataset and Evaluation: The catalog contains 47,071 tracks from 8,975 artists, with organizer-provided embeddings across multiple modalities.
- 2.1 Dataset and Evaluation: Approximately 15,000 training sessions were used, including 500 held out as a development set for hyperparameter tuning.
- 2.1 Dataset and Evaluation: Blind A contains 80 participant-visible sessions for iterative ablations, while Blind B contains 80 sessions held out by the organizers.
- 2.1 Dataset and Evaluation: Table 1 summarizes the dataset statistics used to characterize the challenge data.
- 2.1 Dataset and Evaluation: Sessions are routed as ultra-cold, cold, or warm according to user-ID availability and conversational listening history.
- 2.2 Composite Scoring: The official composite score combines nDCG@20, normalized Judge quality, Distinct-2 lexical diversity, and catalog diversity.
- 2.2 Composite Scoring: Each +1.0 Judge increase contributes +0.075 to the composite score, making response quality materially relevant alongside retrieval.
3 Proposed Pipeline
The submitted pipeline combines multimodal and lexical retrieval, lightweight reranking, and persona-diversified generation, while development experiments test additional signals and models on Blind A.
- 3 Proposed Pipeline: The pipeline has three stages: multimodal retrieval, multi-signal reranking, and response generation, with submitted and development-only components separated.
- 3.1 Stage 1: Multi-Modal Retrieval: Seven dense embedding spaces are supplemented by BM25 and artist matching, with cold sessions using pseudorelevance feedback when history is unavailable.
- 3.1 Stage 1: Multi-Modal Retrieval: A decay factor of γ=0.85 weights recent listening history more heavily while retaining accumulated context.
- 3.1 Stage 1: Multi-Modal Retrieval: Differential-evolution tuning improved MRR from 0.1806 to 0.2158 (+19.5%), with user-CF and artist matching dominating the learned weights.
- 3.2 Submitted Reranking: The submitted reranking applies history filtering, popularity smoothing, and a catalog-diversity penalty to retrieved candidates.
- 3.2 Development-Only Components: Album continuation improved Blind A nDCG from 0.277 to 0.388 (+40%) but was withheld because the pattern might reflect challenge-data artifacts.
- 3.2 Development-Only Components: LambdaMART further improved Blind A nDCG from 0.388 to 0.466 (+20%) but was omitted because of deployment complexity.
- 3.2 Development-Only Components: Conservative GPT-4.1 injection on 9 sessions improved nDCG from 0.4659 to 0.4694 (+0.8%), while broader use was excluded from Blind B.
4 Results and Analysis
The evaluation compares submitted and development-only configurations across leaderboard results, cumulative ablations, and response strategies. Deployment simplifications and split differences complicate direct interpretation of the observed performance gap.
- Official Results: The submitted system’s Blind B leaderboard performance is reported in Table 5, while development alternatives are evaluated separately.
- Ablation Analysis: +41% and +131% were the two largest incremental gains in the Blind A cumulative ablation, from modality expansion and artist substring-match bonus, respectively.
- Response Generation: Table 7 compares response-generation strategies under different retrieval configurations, so its Composite values reflect both response and retrieval quality.
- Deployment Simplifications: GPT-4o-mini Personas replaced GPT-4.1 Mirroring because of cost constraints; the Judge difference was 4.95 versus approximately 2.60, corresponding to an estimated −0.176 composite penalty.
- Deployment Simplifications: Omitting album continuation, XGBoost LTR, and conservative GPT injection produced an estimated −0.096 retrieval penalty; together, simplifications accounted for approximately −0.272 on Blind A.
- Error Analysis: The observed Blind B gap was 0.319, larger than the estimated simplification penalty, suggesting distributional differences between Blind A and Blind B also contributed.
5 Discussion
The discussion finds that LLM-guided artist injection is highly sensitive to intervention scope, while behavioral signals receive strong fusion weights. It also identifies composite-metric trade-offs and scalability constraints.
- Conservative Injection Principle: The 9-session ≤5-artist-track condition yields the best observed Blind A result, while broader intervention degrades nDCG and 54-session injection causes a −18.9% regression.The authors present this as a development-set observation and note that it may not generalize.
- Signal Importance: User collaborative filtering receives a weight of 4.845, exceeding the combined weight of the three Qwen3 text modalities, whose total is 2.534.The optimized weights suggest behavioral signals dominate content-based signals, while user-CF is unavailable in cold-start sessions.
- Composite Metric Navigation: Independent optimization of retrieval nDCG can produce poor response quality and net-negative composite scores, motivating separate retrieval and generation stages.The trade-off arose when GPT-4o rewrote responses while retrieval was aggressively optimized.
- Scalability Considerations: Brute-force cosine retrieval is tractable for 47,071 tracks on the blind set but requires approximate nearest-neighbor search for production catalogs with millions of tracks.GPT API latency remains the primary bottleneck at approximately 1–2 seconds per session.
6 Related Work
The related work positions conversational recommendation as an open-ended, multi-objective setting where LLMs, multimodal fusion, and retrieval-grounded generation address complementary challenges. The paper connects these developments to the TalkPlayData Challenge’s joint retrieval and response-quality objective.
- Conversational and LLM-based Recommendation: Conversational recommender systems use LLMs as interactive agents for open-ended interactions involving subjective attributes and multiple objectives.The literature also reports severe position bias for LLM zero-shot ranking on large candidate sets.
- Multi-Modal Retrieval and Fusion: Modern recommendation integrates dense semantic, audio-text, visual, lexical, and collaborative signals through weighted RRF with differential-evolution tuning.The cited modalities include Qwen-style semantic matching, CLAP, SigLIP, BM25, and BPR.
- Reranking and Retrieval-Augmented Generation: Retrieval-augmented generation grounds conversational responses in retrieved catalog entities, reducing hallucination risk while supporting natural-language justifications.This links retrieval refinement with the generation requirements of conversational systems.
- The TalkPlayData Challenge: The TalkPlayData Challenge extends conversational recommendation to multi-turn interactions with a dual objective of retrieval accuracy and generative response quality.The paper’s multimodal fusion and constrained generation pipeline addresses both objectives.
7 Conclusion
The paper presents a three-stage conversational music recommendation pipeline and reports that constrained LLM intervention outperforms broader injection in the observed Blind A analysis. The submitted system achieves a Blind B composite of 0.321.
- Conclusion: The submitted system combines multimodal RRF retrieval, lightweight reranking, and persona-diversified GPT-4o-mini response generation.Development-only experiments added artist injection, album continuation, LambdaMART, and a GPT-4.1 prompt but were not deployed to Blind B.
- Conclusion: Conservative injection on 9 sessions with ≤5 target-artist tracks reaches Blind A nDCG of 0.4694, while aggressive injection across 54 sessions causes a −18.9% regression.The authors limit this finding to a single evaluation split and call for controlled validation.
- Conclusion: The submitted system records a Blind B composite score of 0.321.