Source-linked AI summary
Ask to Be Sure: Informative Interactions for Confident Multi-Turn LLM Recommendation
Cedar Site Bai, Duanshun Li, Zhenyu Liao, Sheikh Sarwar, Huiyuan Chen, Yuan Chen, Changhe Yuan, Haiyang Zhang, Qilin Qi
TL;DR
Multi-turn conversational recommenders still struggle to elicit user preferences strategically and measure the useful information gained from each interaction. The paper rewards entropy reduction in recommendations to fine-tune LLMs, improving recommendation quality and conversational efficiency on INSPIRED and ReDial.
Problem
Designing multi-turn interactions that effectively elicit user preferences remains an open challenge for conversational recommender systems.
Method
The method measures recommendation uncertainty through entropy reduction and uses it as a reward to select SFT examples or DPO preferences.
Results
Entropy-reduction variants achieve the strongest gains on main conversational metrics across INSPIRED and ReDial, including recommendation hit rates and conversational efficiency.
Takeaways & Limitations
Entropy-based interaction rewards improve recommendation quality and conversational efficiency without requiring ground-truth recommendations.
Takeaways & Limitations
Evaluation relies on automatic metrics and an LLM user simulator with fixed sampling parameters, leaving human-judged usefulness and parameter sensitivity unvalidated.
Abstract
from arXiv · showhide
Recent advances in large language models (LLMs) have enabled their use as conversational recommender systems (CRS), demonstrating strong recommendation accuracy and natural dialogue. However, guiding multi-turn interactions to elicit user preferences effectively remains challenging. Existing approaches either use separate reinforcement learning agents with templated interactions or optimize for interactivity judged by another LLM, without measuring how much useful information is actually gained. We propose a new approach that quantifies the effectiveness of each interaction by the reduction in the assistant's uncertainty, measured via entropy over recommendations. We apply this entropy reduction as a reward---without relying on ground-truth recommendations, which are often unavailable in real-world scenarios---to fine-tune the LLM, enabling strategic interaction generation. Empirical results with supervised fine-tuning (SFT) and direct preference optimization (DPO) on the INSPIRED and ReDial datasets show that our method improves both recommendation quality and conversational efficiency.
1 Introduction
Conversational recommender systems use multiturn natural-language dialogue to discover preferences and refine recommendations, but existing methods struggle to elicit useful information naturally. This work rewards interactions that reduce recommendation uncertainty, enabling strategic generation without ground-truth recommendations.
- Motivation: Conversational recommender systems transform recommendation from one-shot prediction into interactive preference discovery through multiturn dialogue.They can elicit preferences, clarify ambiguous requests, and refine recommendations using real-time feedback.
- Challenges: LLM-based conversational recommenders improve recommendation accuracy and user-aware dialogue, yet preference elicitation and candidate retrieval remain challenging.Other open challenges include integrating user history and collaborative filtering and maintaining up-to-date candidate retrieval.
- Limitations of prior work: Existing preference-elicitation methods often use separate RL agents and rigid multiple-choice or Yes/No interactions, limiting natural conversational flow.In these approaches, LLMs generate dialogue but do not perform recommendation.
- Limitations of prior work: LLM-judge rewards can increase interactivity, but their recommendation-specific information gain and effect on recommendation quality remain unclear.The unresolved issue is how much useful preference information users provide through these interactions.
- Proposed approach: The proposed method rewards each interaction by the reduction in recommendation entropy, using the resulting information gain to fine-tune strategic interaction generation without ground-truth recommendations.The approach is motivated by evidence that lower predictive uncertainty over ranking lists correlates with better recommendation performance.
2 Related Work
Prior work identifies brittleness in multi-turn LLM interaction and develops methods for clarification, preference following, and multi-turn RLHF. It also promotes proactive assistance and recommendation control, while noting that apparent interactivity may not yield useful preference information.
- LLMs remain brittle in multi-turn interaction, motivating methods for clarification, preference following, and multi-turn RLHF.
- CollabLLM encourages proactive assistants using collaboratively generated data and LLM-judge rewards.
- In recommendation contexts, responses can appear interactive while revealing little useful preference information.
- Recommendation-oriented LLM work improves item control, tokenization, and retrieval.
3 Proposed Method
The method quantifies recommendation uncertainty through entropy over sampled recommendation lists and rewards interactions that reduce this uncertainty. It uses turn-level and conversation-level entropy reduction to guide SFT and DPO toward more informative interactions.
- 3.1.1 Entropy over recommendations: Recommendation uncertainty is estimated as Shannon entropy over an empirical item distribution obtained by repeatedly sampling top-m recommendation lists from a fixed conversation prompt.The method samples n lists, aggregates unique recommendations and their empirical frequencies, and computes entropy over the resulting distribution.
- 3.1.1 Entropy over recommendations: Ranking order is incorporated with logarithmic-decay weights, assigning each rank position j the weight w_j=1/log2(j+1).The weighted empirical distribution and weighted entropy account for differences in recommendation rank across sampled lists.
- Entropy reduction: Entropy reduction after an assistant interaction and the elicited user response serves as a reward for recommendation-relevant information gain rather than an LLM judge’s general interactivity assessment.This reward assigns credit to interactions according to their effect on recommendation uncertainty.
- Entropy reduction: The method evaluates both turn-level reduction between consecutive conversation states and conversation-level reduction from an interaction through the terminal unrolled conversation.Turn-level compares H_w(C1) and H_w(C2), whereas conversation-level compares H_w(C1) with H_w(C) after unrolling subsequent turns.
- Entropy reduction: −0.040 bits per turn is the weighted entropy trend across 300 INSPIRED conversations, with mean recommendation entropy generally decreasing as dialogues progress.A single INSPIRED conversation shows the same decreasing entropy pattern as the assistant gathers more user information.
- Optimization: The entropy-reduction reward selects high-reward SFT examples and determines the preferred generation between two DPO samples.Standard SFT and DPO are used as optimization mechanisms, while entropy reduction supplies the reward signal.
4 Experiments
Experiments on INSPIRED and ReDial compare vanilla, SFT, and reward-based fine-tuning under matched data budgets. Entropy-reduction rewards improve recommendation quality and conversational efficiency, outperforming CollabLLM’s reward in simulated conversational recommendation without ground-truth recommendations.
- Datasets and baselines: Baselines include a vanilla zero-shot recommender and supervised fine-tuning, while comparisons use matched data budgets and shared generated preference pairs across reward variants.The experiments do not directly compare with RL methods because their restricted action spaces and interfaces are not directly comparable.
- Datasets and baselines: Experiments use the INSPIRED and ReDial conversational recommendation datasets, with 801 training and 99 test conversations in INSPIRED and 8,631 ReDial training conversations for SFT (Raw).INSPIRED recommendations are user-accepted ground truth; ReDial ground truth uses liked but unseen movies.
- Results: Entropy-reduction variants show the strongest conversational gains: DPO (Turn Entropy) leads INSPIRED Hit@1, Hit@5, and simulated conversation hit rate, while DPO (Conv Entropy) leads ReDial simulated conversation hit rate and turn count.Models use LoRA fine-tuning, temperature 0.1, and averages over three runs.
- Results: Entropy-reduction fine-tuning improves recommendation accuracy and conversational efficiency, requiring fewer turns to recommend the ground truth.The results are consistent with lower recommendation uncertainty being associated with better performance.
- Results: For simulated conversational recommendation, entropy-reduction rewards outperform CollabLLM’s ground-truth-and-LLM-interactivity reward without access to ground-truth recommendations.The reward relies on observed entropy reduction rather than a manually defined notion of good question styles.
- Limitations: Evaluation relies on automatic metrics and an LLM-based user simulator, while conversational usefulness, sampling sensitivity, and retrieval requirements for dynamic catalogs remain unvalidated or necessary.The evaluation fixes m = n = 5 and assumes the base LLM can name candidate items.
5 Conclusion and Future Work
The paper introduces an uncertainty-driven approach that rewards entropy-reducing turns, aligns interaction design with information gain without ground-truth targets, and improves recommendation quality and conversational efficiency on INSPIRED and ReDial.
- Conclusion: The approach quantifies assistant uncertainty as entropy over sampled recommendation lists and rewards interactions that reduce this entropy.This objective uses uncertainty reduction to measure interaction effectiveness.
- Conclusion: The entropy-reduction objective aligns interaction design with information gain, avoids reliance on ground-truth targets, and integrates with SFT and DPO.The method is designed for settings where ground-truth recommendations may be unavailable.
- Conclusion: Fine-tuned models improve recommendation quality and conversational efficiency on the INSPIRED and ReDial datasets.The reported empirical evaluation covers both datasets.
GenAI Usage Disclosure
The authors used generative AI for language polishing and LaTeX drafting while retaining responsibility for the manuscript’s content, experiments, claims, and citations.
- Generative AI assisted with language polishing and LaTeX drafting, while the authors remained responsible for all content, experiments, claims, and citations.