PaperlayerView exact revision on arXiv

Source-linked AI summary

TalkPlay-Tools: Conversational Music Recommendation with LLM Tool Calling

Seungheon Doh, Keunwoo Choi, Juhan Nam

arXiv:2510.01698v4cs.IRcs.MMcs.SDeess.AS

TL;DR

A single retrieval method may not identify items that meet operational constraints while reflecting listening history and text queries. The system has an LLM plan tool types, order, and arguments, then orchestrate SQL, BM25, dense retrieval, and Semantic IDs in a retrieval-reranking pipeline. In zero-shot conversational music recommendation, the tool-calling approach achieves Hit@1 0.022, versus 0.018 for Qwen3-LM + BM25.

  • Problem

    A single retrieval method may not identify items that meet operational constraints while reflecting listening history and text queries.

    Source: §1 · p. 1

  • Method

    The system has an LLM plan tool types, order, and arguments, then orchestrate SQL, BM25, dense retrieval, and Semantic IDs in a retrieval-reranking pipeline.

    Source: §1 · p. 2, §2.1 · p. 2

  • Results

    In zero-shot conversational music recommendation, the tool-calling approach achieves Hit@1 0.022, versus 0.018 for Qwen3-LM + BM25.

    Source: §4 · p. 5

  • Takeaways & Limitations

    The framework harnesses text, audio, visual, and behavioral information to deliver music recommendations in conversational contexts.

    Source: §5 · p. 6

Abstract

from arXiv · show

While the recent developments in large language models (LLMs) have successfully enabled generative recommenders with natural language interactions, their recommendation behavior is limited, leaving other simpler yet crucial components such as metadata or attribute filtering underutilized in the system. We propose an LLM-based music recommendation system with tool calling to serve as a unified retrieval-reranking pipeline. Our system positions an LLM as an end-to-end recommendation system that interprets user intent, plans tool invocations, and orchestrates specialized components: boolean filters (SQL), sparse retrieval (BM25), dense retrieval (embedding similarity), and generative retrieval (semantic IDs). Through tool planning, the system predicts which types of tools to use, their execution order, and the arguments needed to find music matching user preferences, supporting diverse modalities while seamlessly integrating multiple database filtering methods. We demonstrate that this unified tool-calling framework achieves competitive performance across diverse recommendation scenarios by selectively employing appropriate retrieval methods based on user queries, envisioning a new paradigm for conversational music recommendation systems.

1 Introduction

Music recommendation has moved from metadata filtering toward embedding-based retrieval, multimodal content understanding, and interactive LLM systems. The paper addresses the limits of relying on a single retrieval method with a tool-calling recommender that combines multiple tools and improves Hit@K over strong baselines.

1 Introduction

  • Music recommendation has progressed from boolean filtering and large-scale embeddings to content understanding, multimodal representations, and interactive LLM-based recommendation.The cited developments support example-based recommendation, natural-language query understanding, generative recommendation, and multi-turn music discovery.Source: §1 · p. 1
  • Production recommenders must satisfy operational constraints while reflecting both listening history and text queries, needs a single retrieval method may not fully capture.The paper identifies combining multiple retrieval methods as necessary to address this gap.Source: §1 · p. 1
  • The paper proposes an LLM that predicts tool types, order, and arguments, combining SQL, BM25, dense retrieval, and Semantic IDs in one system.The authors report zero-shot effectiveness on a conversational recommendation benchmark with improved Hit@K over strong baselines.Source: §1 · p. 2

2 Music Recommendation with Tool Calling

TalkPlay-Tools pairs an LLM-based agent with an external environment that executes tools for retrieval and reranking. The LLM plans sequential tool calls from user and conversation context, drawing on structured, lexical, dense, and generative retrieval tools.

2 Music Recommendation with Tool Calling

  • The framework separates a Music Recommendation Agent comprising the LLM and tools from an External Environment that executes tools and performs final retrieval and reranking.The external environment supplies the execution side of the agent's recommendation process.Source: §2 · p. 2

2.1 Problem Formulation

  • Given a user profile, prior conversation state, and current query, the LLM predicts tool calls that filter the database into ranked music items.Because each tool refines the pool available to the next, the system stages calls as retrieval followed by reranking.Source: §2.1 · p. 2

2.2 Tools

  • SQL filters relational metadata, while BM25 retrieves lexically across title, artist, album, lyrics, and attribute corpora.BM25 is described as useful for text queries with typos or difficult exact-string matches.Source: §2.2 · p. 3
  • Dense tools support text-to-item, item-to-item, and user-to-item retrieval, while Semantic IDs enable generative retrieval through indexed multimodal content codes.Text-to-item handles natural-language descriptions, item-to-item finds similar examples, and user-to-item uses listening-history embeddings.Source: §2.2 · p. 3

2.3 In-Context Information for Inference

  • Inference context includes a three-stage planning, retrieval, and reranking prompt, user demographics and recent listening, and prior conversation history with track information.A separate response-generation prompt explains recommendations in relation to user queries.Source: §2.3 · p. 4

3 Experiments

Experiments use TalkPlayData 2 test conversations and compare tool calling with BM25 and Qwen3-LM + BM25. The supplied experiment passages specify the evaluation setup and identify first-attempt tool-frequency and success-rate analyses.

3 Experiments

  • TalkPlayData 2 supplies multimodal profiles and music representations; evaluation uses 1,000 test conversations with eight turns each.The synthetic dataset includes listening history, metadata, semantic tags, lyrics, album art, and audio.Source: §3 · p. 4
  • The environment combines metadata and genre/style annotations, pretrained text, audio, and image encoders, BPR embeddings, and multimodal Semantic IDs.Chronological splits keep training data before test data for BPR and RVQ models.Source: §3 · p. 4
  • The evaluation compares BM25, Qwen3-LM + BM25, and Qwen3-LM + Tool Calling using Hit@1, Hit@10, and Hit@20.Figures 2 and 3 are identified as analyses of first-attempt tool-calling frequency and success rate; their captions provide no outcome values.Source: §3 · p. 4, Figure 2 · p. 5, Figure 3 · p. 5

4 Results

TalkPlay-Tools outperforms baseline methods in zero-shot multi-turn recommendation, while tool-use analyses show varied invocation frequencies and success rates. A qualitative example illustrates its staged inference process.

Multi-turn Recommendation

  • In zero-shot conversational music recommendation, the tool-calling approach achieves Hit@1 of 0.022 versus 0.018 for Qwen3-LM + BM25, outperforming baseline methods.The passage attributes the improvement to reranking through multiple tool integration.Source: §4 · p. 5

Tool Calling Frequency

  • SQL and BM25 are used more frequently than item-to-item matching and Semantic ID tools, which the authors suggest may reflect pretrained LLMs’ greater familiarity with the former.Figures 2 and 3 analyze first-attempt tool calls; repeated retries eventually yield a result in the experiment.Source: §4 · p. 5

Tool Calling Success Rate

  • SQL succeeds in 27.4% of calls and item-to-item matching in 68.4%, while user-to-item personalization succeeds in 98.8% and Semantic ID retrieval in 95.8%.The paper links SQL errors to syntactic complexity and metadata mismatches, item-to-item errors to predicting complete track IDs, and the latter tools’ high success to profile and dialogue context.Source: §4 · p. 5

Qualitative Example

  • The inference example proceeds from user information and dialogue context through tool selection and retrieval to a natural-language response.Table 3 presents an example of the LLM’s inputs and outputs.Source: §4 · p. 5, Table 3 · p. 6

5 Conclusion

The conclusion presents a multimodal tool-calling framework for conversational music recommendation and identifies future directions for improving tool selection and personalization.

  • The framework integrates multiple modalities and components of a retrieval-reranking pipeline through LLM tool calling with external executors and models.Source: §5 · p. 6
  • It uses text, audio, visual, and behavioral information for music recommendations in conversational contexts.Source: §5 · p. 6
  • Future directions include reinforcement learning to improve tool-calling precision and reduce retries, and personalized strategies extending beyond track-centric approaches.The latter aim is to capture nuanced user preferences and behavioral patterns more effectively.Source: §5 · p. 6

A Tool Calling Functions

The callable tools combine structured, lexical, similarity-based, personalized, and Semantic ID retrieval. Their outputs are track identifiers that can be composed sequentially in the retrieval-reranking pipeline.

  • The callable tools return lists of track_id strings up to topk and are designed for sequential composition in the retrieval-reranking pipeline.The appendix specifies each tool’s signature, purpose, arguments, and expected return.Source: A Tool Calling Functions · p. 9
  • SQL performs boolean matching over track metadata, while BM25 supports lexical search across titles, artists, albums, lyrics, and attributes.The SQL schema includes title, artist, album, popularity, release date, tempo, and key.Source: A Tool Calling Functions · p. 9
  • Text-to-item and item-to-item similarity tools retrieve by modality, while user-to-item similarity provides personalization and is excluded for cold-start users.Supported modalities include text, audio, and image for text-to-item; audio, image, and collaborative filtering for item-to-item.Source: A Tool Calling Functions · pp. 9–10
  • Semantic ID matching retrieves track identifiers from codebook indices across audio, image, metadata, lyrics, attributes, and collaborative-filtering item modalities.Source: A Tool Calling Functions · p. 10
Loading 2510.01698v4…