Source-linked AI summary
When the Label Ignores the Request: Auditing Policy-Selected Targets in Synthetic Conversational Music Recommendation
Sanjeev Suresh
TL;DR
Synthetic conversational-recommendation benchmarks use policy-selected tracks as official labels, which may not match a user's visible request. This paper audits exact-song requests in TalkPlay and tests a guarded supplement of catalog-resolved targets. The supplement improves request satisfaction on conflict turns while preserving the official metric.
Problem
Policy-selected labels in synthetic conversational recommendation record what the generation pipeline played next, not necessarily what the user visibly requested, leaving exact-song label conflicts to be measured.
Source: §1 · p. 1Method
The authors audit visible exact-song requests against catalog metadata and supplement official training targets with resolved request-satisfying targets, evaluating against a matched control.
Source: §1 · pp. 1–2, §6 · p. 3Results
Takeaways & Limitations
Exact-song requests form an auditable slice where policy-selected labels can contradict visible user intent, and request-satisfying targets can improve that slice without replacing official labels.
Source: §1 · pp. 1–2The detector is TalkPlay-specific and abstains when no catalog item resolves, so 41/82 describes audited turns rather than the benchmark as a whole.
Source: §10 · p. 5
Abstract
from arXiv · showhide
Synthetic dialogues generated by LLM pipelines now serve as complete conversational-recommendation benchmarks: an LLM listener talks to an LLM recommender, and the track logged next in the conversation becomes the official label for each turn. These policy-selected labels make large-scale evaluation reproducible, but they are proxies for what the simulated user asked. We audit the one place where label and request are directly comparable: turns where the user asks for an exact song by name. In the RecSys Challenge 2026 TalkPlay benchmark, using visible dialogue and catalog metadata alone, we find that the official label contradicts the user's exact-song request in half of the audited development turns. This matters beyond one benchmark: naming the desired item is the dominant intent in real music search, where deployed systems avoid substituting an alternative for an exactly named item, on the premise that it costs satisfaction. A small training-time supplement closes most of the gap: adding catalog-resolved request-satisfying targets to a small fraction of training turns yields a 53.3% relative gain in nDCG@20 on the 43 conflict turns while leaving the official metric intact, verified against a matched control that detects the same requests but trains only on official labels.
1 Introduction
The introduction audits exact-song turns, where a visible request can be checked against the official label, in the synthetic TalkPlay benchmark. It finds frequent conflicts and proposes a guarded training supplement while preserving the official metric.
1 Introduction
- 41 of 82 audited exact-song turns have an official label that points to a different track than the user's request.The audit uses the organizer development set and resolves requests against catalog metadata.Source: §1 · p. 1
- Exact-song requests matter beyond this benchmark because naming the desired item dominates real music interaction, and deployed systems avoid substitution on focused queries on the premise that it costs satisfaction.The passage cites known-item search, title or performer metadata in queries, and playback as a common voice-assistant command.Source: §1 · p. 1
- The audit deliberately acts only where requests resolve to certifiable targets; broad visible constraints can have unreliable metadata and ambiguous target sets.The paper reports screening and declining five request families.Source: §1 · p. 1
- Supplementing 1.5% of training turns with request-satisfying targets raises conflict-slice nDCG@20 from 0.523 to 0.802 while preserving the official metric, against a matched control.The contribution also includes released annotations and tooling for reproducing the audit and supplement.Source: §1 · p. 2
- The paper treats the finding narrowly: visible exact-song requests reveal a measurable gap, and training can close it without adjudicating which track is correct.Source: §1 · p. 2
2 Policy-Selected Labels and the Audited System
TalkPlayData 2 logs the recommender's selected continuation as the official label, even though that choice can reflect hidden session context rather than the user's latest request. The paper's system ranks the full catalog using a retrieval-and-reranking cascade and visible dialogue and catalog information.
2 Policy-Selected Labels and the Audited System
- The official label is the pool track selected after a user turn, but the pool is limited to 16–32 tracks drawn from a real user's listening session.The Listener LLM is conditioned on a session goal and inferred profile; repeats are forbidden, and the pool excludes profiling tracks.Source: §2 · p. 2
- A policy-selected label can reflect the session goal, dialogue flow, or inferred profile rather than the latest visible request; an out-of-pool request forces a substitute.Such conflicts need not be annotation errors because the generator serves its own goal under its constraints.Source: §2 · p. 2
- The challenge ranks the full catalog, while the recommendation pool and some profile attributes remain internal; the paper computes its results from visible dialogue and catalog metadata alone.Source: §2 · p. 2
- The challenge system retrieves candidates through lexical, dense, feedback, and dialogue-to-track bi-encoder arms, then scores their union with an XGBoost LambdaRank reranker.The deduplicated union contains about 1,550 candidates per turn; the final submission adds session co-occurrence features and score diffusion.Source: §2 · p. 2
3 Related Work
Related work identifies validity and evaluation problems in conversational recommendation, but this paper audits a distinct case: a visible request resolves to a catalog item that conflicts with the logged policy choice. It positions the audit alongside behavioral testing and work on exact-item instruction following.
3 Related Work
- Prior CRS work highlights static-label limitations, target leakage, repeated recommendations, and disagreement about gold items; this audit instead measures visible-request conflict in a synthetic benchmark.The paper characterizes this as complementary to leakage, the opposite failure mode.Source: §3 · p. 2
- Unlike re-annotation, counterfactual learning, or relevance expansion, the proposed intervention supplements rather than removes the official label and uses catalog resolution of the visible request.The paper requires the supplement to beat a matched control and preserve the official metric.Source: §3 · p. 2
- Behavioral test suites expose failures beyond aggregate accuracy, while this work also audits benchmark labels; prior intent catalogs did not distinguish named-track requests from genre or mood filters.Source: §3 · pp. 2–3
4 Audit Design
The audit acts only when visible evidence supports a reliable request target, and any intervention supplements rather than replaces official supervision. A three-part evidence bar governs whether a request family qualifies for action.
4 Audit Design
- Detection and deployment-time actions use visible dialogue and catalog metadata, never hidden reasoning traces, and the official label is not an input to detection.Source: §4 · p. 3
- An intervention may add the visibly requested target or compare it with the official label, but it never removes the official label or adjudicates which item is correct.Source: §4 · p. 3
- A request family must resolve to a concrete target set with ambiguity abstention and strict detector precision, beat a matched control, and improve request satisfaction within the official-metric preservation margin.The paper reports five visible families that failed this evidence bar.Source: §4 · p. 3
5 The Exact/Version Request Family
The exact/version family covers visible requests for a named song or equivalent version, resolved to catalog tracks. On the development set, the audit finds conflicts in half of the detected directives.
5 The Exact/Version Request Family
- The admitted family includes a specific song title or a version/duplicate equivalent, such as the same recording under an alternate catalog entry.The detector extracts titles and request spans, then resolves them through catalog title and artist fields; it abstains on ambiguity or broad preferences.Source: §5 · p. 3
- 41 of 82 detected exact directives have an official label that names a different track from the request.The 82 directives are 1.0% of 8,000 evaluated development turns.Source: §5 · p. 3
- In a stratified training-split audit, 37 of 40 detector positives were explicit track requests with the same extracted title as the annotator.The annotation pass also found three requests among 80 turns the detector did not flag; the audit prioritizes precision, since false fires could inflate the conflict rate.Source: §5 · p. 3
6 Training with Request-Satisfying Targets
The intervention adds catalog-resolved request-satisfying targets only when an exact/version request conflicts with the official label, while retaining official supervision. A matched control receives the same request feature but trains only on official labels.
6 Training with Request-Satisfying Targets
- The supplement affects 1,873 of 121,592 training turns, retaining official labels and adding request-satisfying targets at a reduced group weight of 0.1.Turns without a conflict remain untouched; 13 affected turns add no rows because the resolved track is absent from the candidate pool.Source: §6 · p. 3
- The matched control shares the specialist’s features, candidate pools, gate, schedule, and evaluation, but trains on full-weight official labels only.This comparison tests whether the gain comes from request-satisfying targets rather than merely detecting an exact request.Source: §6 · p. 3
- The evaluation slice contains 43 directives where the resolved request target and official label diverge; the configuration was frozen before full-split retraining and evaluation.The slice includes 42 exact-track cases and one version/duplicate case.Source: §6 · pp. 3–4
7 Results
The specialist improves request satisfaction on the conflict slice while preserving the official whole-split metric. Its gains are concentrated on turns where the visible request and policy-selected label diverge.
7 Results
- Across all 82 audited directives, the specialist ranks the requested track first on 66, versus 46 for the control, and ranks it higher on 35 turns and lower on none.The table compares requested-track ranks by label status under the two gated models.Source: §7 · p. 4
- 53.3% relative gain: conflict-slice nDCG@20 rises from 0.523 to 0.802, while the official whole-split delta is +0.0006 (+0.31%).The official-metric confidence interval is [+0.0002, +0.0012], within the predeclared preservation margin and excluding loss.Source: §7 · p. 4
- Request-satisfying nDCG@20 improves by 1.09% on the whole development split, with a 95% CI of [+0.0011, +0.0032].The readout credits either the official label or resolved request targets on audited turns.Source: §7 · p. 4
- The large slice gain and small whole-split changes align with the intervention’s 1.5% training-turn coverage and the 43-case conflict slice.The matched control rules out mere exposure to the exact-request feature as an explanation for the specialist’s gain.Source: §7 · p. 4
8 Other Request Families
Exact/version requests are the only visible request family the study acts on; five other families fail the evidence bar for target resolution, precision, or preservation.
8 Other Request Families
- Artist constraints fall short on precision, while suppressing rejected tracks causes a confirmed official-metric loss without request-satisfaction gain.The artist detector reached 74% action precision; switch-away turns also lack enough admissible alternatives for a clean top-20.Source: §8 · p. 4
- Album and year constraints fail target resolution because album matching is fuzzy and release-date fields conflate original and reissue dates.Broad semantic requests fail certifiability: none of 125 screened requests yielded a catalog-verifiable target set.Source: §8 · p. 4
- The study declines other families unless their evidence clears the full bar for acting on a request family.Exact/version requests are the only family admitted for training.Source: §8 · p. 4
9 Reproducibility and Artifact
The section documents the released audit and training artifacts and distinguishes the challenge leaderboard result from the training supplement.
9 Reproducibility and Artifact
- Released artifacts include ID-keyed annotations, the exact/version detector, target-remap script, specialist and matched-control training scripts, and a hashed evidence manifest.The manifest records paths and hashes for the evidence package.Source: §9 · p. 4
- The Blind-B leaderboard submission scored composite 0.528 and nDCG@20 0.440, ranking 12 of 44; the Section 6 supplement was not included.Source: §9 · pp. 4–5
- The code repository is linked for access to the released materials.Source: §9 · p. 5
10 Conclusion
The conclusion reports improved conflict-slice performance while limiting the audit rate to turns with catalog-resolvable exact-song requests. It proposes porting the audit and checking for contradictory labels during generation.
10 Conclusion
- Supplementing 1.5% of training turns with request-satisfying targets raises conflict-slice nDCG@20 from 0.523 to 0.802, beats a matched control, and preserves the official metric.Source: §10 · p. 5
- Official labels point away from the requested track in half of 82 audited turns, but exact directives are only 1.0% of evaluated turns, so the whole-split average barely moves.Source: §10 · p. 5
- Because the detector abstains when no catalog item resolves, 41/82 describes audited turns rather than the benchmark at large.Source: §10 · p. 5
- The authors propose porting the audit to other synthetic conversational-recommendation benchmarks and checking contradictory labels inside the generation loop.The detector itself is TalkPlay-specific; annotations and tooling are released so the slice can be reported beside the official metric.Source: §10 · p. 5