Source-linked AI summary
Unified Conversational Recommendation Policy Learning via Graph-based Reinforcement Learning
Yang Deng, Yaliang Li, Fei Sun, Bolin Ding, Wai Lam
TL;DR
Existing CRS methods often separate decisions about asking attributes, recommending items, and timing, limiting generality and stable training. This paper unifies those decisions in a dynamic weighted graph-based RL policy with preference- and entropy-based action selection. The method significantly outperforms state-of-the-art CRS methods across benchmark datasets and a real-world E-Commerce application, while improving scalability and stability.
Problem
Existing CRS methods mainly address subsets of asking, recommending, and timing with separated components, restricting generality and stable training.
Method
UNICORN formulates the three decisions as unified policy learning using a dynamic weighted graph and two strategies that reduce the candidate action space.
Results
UNICORN significantly outperforms state-of-the-art CRS methods across four benchmark datasets and a real-world E-Commerce application.
Takeaways & Limitations
The unified framework demonstrates scalability and stability for conversational recommendation across benchmark and real-world settings.
Abstract
from arXiv · showhide
Conversational recommender systems (CRS) enable the traditional recommender systems to explicitly acquire user preferences towards items and attributes through interactive conversations. Reinforcement learning (RL) is widely adopted to learn conversational recommendation policies to decide what attributes to ask, which items to recommend, and when to ask or recommend, at each conversation turn. However, existing methods mainly target at solving one or two of these three decision-making problems in CRS with separated conversation and recommendation components, which restrict the scalability and generality of CRS and fall short of preserving a stable training procedure. In the light of these challenges, we propose to formulate these three decision-making problems in CRS as a unified policy learning task. In order to systematically integrate conversation and recommendation components, we develop a dynamic weighted graph based RL method to learn a policy to select the action at each conversation turn, either asking an attribute or recommending items. Further, to deal with the sample efficiency issue, we propose two action selection strategies for reducing the candidate action space according to the preference and entropy information. Experimental results on two benchmark CRS datasets and a real-world E-Commerce application show that the proposed method not only significantly outperforms state-of-the-art methods but also enhances the scalability and stability of CRS.
1 INTRODUCTION
CRS must jointly decide what attribute to ask, which items to recommend, and when to ask or recommend. The paper unifies these decisions in a dynamic graph-based RL policy with action-space reduction for more scalable and stable learning.
- Motivation: CRS extends item recommendation by also deciding what questions to ask and when to ask or recommend.These decisions coordinate conversation and recommendation during multi-round interactions.
- Challenges: Existing CRS methods separate conversation and recommendation components, limiting generality and making policy learning harder to converge.They may require offline recommendation models or synthetic dialogue histories.
- Unified formulation: The paper formulates when to ask or recommend, what to ask, and which items to recommend as one unified policy-learning problem.The unified policy selects either an attribute query or item recommendation at each turn to maximize cumulative utility.
- Method: UNICORN uses a dynamic weighted graph and graph-based MDP to model changing relationships among users, items, and attributes during conversation.The framework is designed to combine conversation and recommendation decisions within one policy.
- Method: Preference-based item selection and weighted entropy-based attribute selection reduce the candidate action space to address sample-efficiency challenges.The strategies avoid enumerating the entire item and attribute sets.
- Results: Experimental results show significant improvements over state-of-the-art CRS methods across four public datasets and a real-world E-Commerce application.The reported gains include improved scalability and stability.
2 RELATED WORKS
Related CRS research spans cold-start exploration, question asking, dialogue understanding and generation, and multi-round recommendation. This paper focuses on multi-round conversational recommendation, where systems alternate between asking about attributes and recommending items.
- Exploration-exploitation: CRS research includes bandit approaches that balance exploration and exploitation for cold-start users.
- Question-driven approaches: Question-driven approaches ask users questions to obtain additional preference information.
- Dialogue understanding and generation: Dialogue understanding and generation methods interpret user preferences and intentions while producing fluent responses.
- Multi-round conversational recommendation: Multi-round conversational recommendation repeatedly asks about attributes or makes recommendations to achieve successful recommendations with fewer conversation turns.
3 PROBLEM DEFINITION
In the multi-round conversational recommendation setting, a system repeatedly asks about attributes or recommends items while updating candidate sets from user responses. The objective is successful recommendation with the minimum number of interactions.
- Setting: The MCR setting allows the CRS to ask attribute questions or make item recommendations multiple times.
- Interaction process: Each episode starts with a user specifying an attribute, after which the system selects an attribute question or top-K items from candidate sets.
- Interaction process: Users accept or reject queried attributes and recommended items, and the CRS updates candidate attribute and item sets from those responses.
- Reinforcement-learning formulation: The CRS formulates the interaction as a multi-step decision process in which the state reflects system status and conversation history, the action is the agent decision, and the reward is intermediate feedback.
4 METHODOLOGY
UNICORN models unified conversational recommendation as a graph-based MDP that selects attribute-asking or item-recommending actions. It combines graph and conversation representations, prunes candidate actions, and trains a deep Q-learning policy.
- Graph-based MDP Environment: UNICORN represents conversational recommendation with a graph-based MDP whose environment exposes states, actions, transitions, and rewards.The state includes conversation history and a graph containing users, items, and attributes.
- Graph-based MDP Environment: At each turn, the agent selects either a candidate item to recommend or a candidate attribute to ask.Candidate sets are updated after user responses, which drive subsequent state transitions.
- Graph-enhanced State Representation: A dynamic weighted graph represents changing relationships among users, items, and attributes during the conversation.Item edge weights incorporate user preference and correlations between asked attributes and items.
- Graph-enhanced State Representation: Graph convolutional and Transformer-based modules combine graph structure with sequential conversation information into a learned state representation.Graph convolution refines node representations using neighboring nodes, while the Transformer models conversation history.
- Action Selection Strategy: Preference-based item selection and weighted entropy-based attribute selection reduce the candidate action space for more sample-efficient learning.The method selects top-K_v candidate items and top-K_p candidate attributes before policy learning.
- Deep Q-Learning Network: The policy uses deep Q-learning with a dueling Q-network and prioritized replay to support unified policy learning.The training procedure is presented as Algorithm 1 and includes candidate-action construction before learning.
5 EXPERIMENT
UNICORN is evaluated across benchmark and real-world conversational recommendation datasets using success rate, average turns, and hierarchical discounted gain. It consistently outperforms baselines, trains more stably, and benefits from graph, sequential, pretraining, and action-selection components.
- Experimental setup: UNICORN is evaluated on four benchmark datasets and a real-world E-Commerce platform using conversational recommendation experiments.The datasets include LastFM, Yelp, LastFM*, and Yelp*.
- Overall performance: UNICORN significantly outperforms all baselines in success rate, average turn, and hDCG across five datasets.On the E-Commerce dataset, it achieves margins of about 18% for SR@15, 2 turns for AT, and 30% for hDCG.
- Training efficiency: UNICORN trains more stably and reaches better performance with fewer training episodes than strong baselines.The comparison uses test-performance curves across training episodes.
- Performance across conversation turns: UNICORN consistently outperforms baselines across nearly every conversation turn and is strongest in the middle stage with many candidate actions.Its middle-stage performance indicates scalability to large candidate action spaces, while SCPR becomes closer later as the candidate set shrinks.
- Ablation study: Sequential representation learning, pretrained graph embeddings, and both preference- and entropy-based action selection materially support performance.Removing sequential learning causes larger decreases on four of five datasets; random or non-graph-pretrained embeddings perform worse, and pruning actions is necessary for large search spaces.
- Qualitative analysis: In a sampled E-Commerce interaction, UNICORN avoids redundant questioning and integrates recommendation and conversation decisions more comprehensively than competing methods.CRM mainly recommends, EAR asks dispreferred questions, and SCPR can ask unnecessary questions because it only decides when to ask or recommend.
6 CONCLUSIONS
The paper unifies three CRS decisions into one dynamic weighted graph-based RL framework and adds action-selection strategies for sample efficiency. Across benchmark datasets and a real-world E-Commerce application, the method outperforms state-of-the-art CRS methods with improved scalability and stability.
- UNICORN unifies when to ask or recommend, what to ask, and which items to recommend into one policy-learning problem.
- The proposed adaptive reinforcement-learning framework uses a dynamic weighted graph to address unified conversational recommendation policy learning.
- Two action-selection strategies are introduced to address sample-efficiency issues.
- The method significantly outperforms state-of-the-art CRS methods across four benchmark datasets and a real-world E-Commerce application.
- The experiments show remarkable scalability and stability for the proposed method.