Unified Conversational Recommendation Policy Learning via Graph-based Reinforcement Learning

1 INTRODUCTION

• CRS must choose what to ask, which items to recommend, and when to ask or recommend, while MCR seeks successful recommendations with minimal interaction.The paper studies multi-round conversational recommendation, in which the system asks about preference attributes or recommends items across multiple turns.Source: §1 · p. 1
• Existing CRS methods split these decisions across components, limiting generality and making policy learning hard to converge.Some methods learn when and what attributes to ask while relying on an external recommendation model; another learns when to ask or recommend but leaves what and which decisions to isolated components.Source: §1 · pp. 1–2
• The paper formulates all three CRS decisions as a unified policy that chooses an attribute query or item recommendation each turn to maximize cumulative utility.The motivation is to coordinate recommendation and conversation decisions during training.Source: §1 · p. 2
• UNICORN uses a dynamic weighted graph to integrate conversation and recommendation, with preference-based item selection and weighted entropy-based attribute selection to narrow candidate actions.The graph models changing relationships among users, items, and attributes; graph-enhanced representations and sequential conversation modeling capture dynamic user preferences.Source: §1 · p. 2
• The introduction reports that UNICORN significantly outperforms state-of-the-art CRS methods across four public benchmark datasets and a real-world E-Commerce application.Source: §1 · p. 2

Conversational Recommendation

• CRS work spans cold-start exploration–exploitation [40], question-driven methods, dialogue understanding and generation [16] [18] [47] [48], and multi-round recommendation [12] [26].The paper focuses on multi-round conversational recommendation, where systems ask preference questions or recommend multiple times to achieve engaging, successful recommendations in fewer turns.Source: §2 · pp. 2–3

RL in Recommendation

• RL recommendation typically learns which items to recommend, while CRS applies RL to which attributes to ask [12] [26] and when to ask or recommend.Because RL models user-agent interactions as an MDP and can account for long-term feedback [44], this work unifies all three CRS decisions in one policy-learning problem.Source: §2 · p. 3

Graph-based Recommendation

• Graph-based recommendation uses graph representations to improve performance or path reasoning to support explainability, and graph-based RL has been applied across recommender scenarios [33] [36] [49].Prior examples include policy-guided path search [36] and GCN-based state representations [10] [49]; this work studies graph-based RL for conversational recommendation using a dynamic weighted graph.Source: §2 · p. 3

Multi-round Conversational Recommendation

• MCR lets the CRS ask about attributes or recommend items across multiple turns, ending when it reaches the target item or the turn limit T [12, 14].The user begins by specifying an attribute and responds to system actions by accepting or rejecting queried attributes or recommended items; the CRS updates candidate sets after each response.Source: §3 · p. 3

Unified Conversational Recommendation Policy Learning

• The MCR task is formulated as an MDP in which the CRS selects an attribute query or item recommendation from past interactions and learns a policy to maximize expected cumulative rewards.The interaction repeats until the CRS hits the user-preferred items or reaches the maximum turn T.Source: §3 · p. 3
• The state s_t represents the system state and conversation history, a_t is the action at timestep t, and r_t is the intermediate reward.Source: §3 · p. 3

• Figure 3 depicts UNICORN’s Graph-based MDP Environment, Graph-enhanced State Representation Learning, Action Selection Strategy, and Deep Q-Learning Network.Source: §4 · p. 3

4.1 Graph-based MDP Environment

• The MDP environment is defined by state, action, transition, and reward components.It informs the agent about the current state and possible actions, then rewards it according to how its policy fits observed user interactions.Source: §4.1 · p. 3

4.1.1 State.

• Each state includes prior conversation history and the full user-item-attribute graph, alongside a user-specific dynamic subgraph and preference and rejection information.The initial state is initialized from the user-specified attribute.Source: §4.1.1 · p. 3

4.1.2 Action

• At each turn, the agent chooses an action from candidate items to recommend or candidate attributes to ask about.The candidate item set contains items satisfying preferred attributes, and the candidate attribute set contains attributes connected to at least one candidate item.Source: §4.1.1 · p. 4

4.1.3 Transition.

• User responses drive state transitions, with accepted attributes added to the preferred set; the environment assigns five rewards for recommendation success or failure, attribute acceptance or rejection, and reaching the turn limit.Recommendation acceptance and reaching the maximum number of turns receive strongly positive and strongly negative rewards, respectively; accepting an asked attribute receives a slightly positive reward.Source: §4.1.3 · p. 4, §4.1.4 · p. 4

4.2 Graph-enhanced State Representation

• Graph-based pre-training produces node embeddings for all nodes in the full graph G, capturing interrelationships among users, items, and attributes.Source: §4.2 · p. 4

4.2.1 Dynamic Weighted Graph Construction.

• The current state is represented as an undirected dynamic weighted graph, with item recommendation scores and user preference and attribute–item correlations informing its edge weights.The weight uses user, item, and attribute embeddings and a sigmoid function.Source: §4.2.1 · p. 4

4.2.2 Graph-based Representation Learning.

• A graph convolutional network refines node representations using structural and relational knowledge from graph connectivity.The update aggregates neighboring-node and self-node information using trainable transformations and a normalized adjacency matrix.Source: §4.2.2 · p. 4

4.2.3 Sequential Representation Learning.

• A Transformer encodes accepted attributes in conversation order to capture history and attend to information relevant to the next action, then mean pooling combines graph and history information into the state representation.The Transformer input is the sequence of accepted attributes P(t).Source: §4.2.3 · p. 5

Preference-based Item Selection

• Preference-based item selection limits recommendations to the top-K_v items that best fit the user’s preferences, adding them to the candidate action space at each timestep.The method assumes users are unlikely to be interested in all items.Source: §4.3 · p. 5

Weighted Entropy-based Attribute Selection

• Weighted entropy prunes candidate attributes by favoring those expected to reduce uncertainty about candidate items while encoding user preference.The method selects the top-K_p candidate attributes.Source: §4.3 · p. 5

4.4.1 Dueling Q-Network.

• DQN learns the unified conversational recommendation policy, while dueling Q-networks estimate value and advantage with separate neural networks to form Q-values.The Q-value represents expected reward from a state and action under discounted rewards.Source: §4.4 · p. 5, §4.4.1 · p. 5

4.4.2 Double Q-Learning with Prioritized Experience Replay.

• The agent uses ε-greedy sampling to balance exploration and exploitation, stores feedback-based transitions in a replay buffer, and trains DQN from sampled mini-batches.The candidate action space is updated after each user response; the loss uses a target value based on the currently optimal Q-function.Source: §4.4.2 · pp. 5–6
• Double Q-learning uses a periodically copied target network to address conventional DQN’s overestimation bias, with soft assignment updates controlled by τ.The target network is a periodic copy of the online network.Source: §4.4.2 · p. 6
• Prioritized replay [24] samples transitions with probability δ relative to absolute TD error, favoring experiences with greater learning potential over uniform sampling.Source: §4.4.2 · p. 6
• At inference, the system asks about a selected attribute or recommends the top-K items with the highest Q-values.Given the user and conversation history, it obtains the candidate action space and current state representation before choosing the next action.Source: §4.4.3 · p. 6

5.1 Dataset

• The evaluation uses LastFM and Yelp in both grouped-attribute and original-attribute versions, plus a Taobao Women_Dress dataset with 30,000 sampled items.The Taobao dataset treats concatenated product properties and values as attributes and includes purchased_by and belong_to graph relations.Source: §5.1 · p. 6

5.2 Experimental Settings; 5.2.1 User Simulator; 5.2.4 Implementation Details

• Sessions simulate each observed user-item pair, using the target item’s attributes as preferred attributes and an initial randomly chosen preference.E-Commerce uses a 7:1.5:1.5 train/validation/test split, 10-item recommendation lists, 15 maximum turns, and 10,000 online-training episodes; the embedding and hidden sizes are 64 and 100.Source: §5.2.1 · p. 6, §5.2.4 · p. 7

5.2.2 Baselines

• Baselines include Max Entropy, Abs Greedy, CRM, EAR, and SCPR, spanning rule-based, recommendation-only, and conversational policy approaches.SCPR uses graph path reasoning to prune candidate attributes and DQN to decide when to ask or recommend.Source: §5.2.2 · p. 6

5.3 Performance Comparison; 5.3.1 Overall Performance

• UNICORN outperforms all baselines across five datasets, with gains of about 18% in SR@15, 2 turns in AT, and 30% in hDCG.On E-Commerce, SCPR surpasses EAR and CRM, but remains limited by separated conversation and recommendation components; UNICORN combines mutually enhancing components with action-selection strategies for sample efficiency.Source: Table 2 · p. 7, §5.3.1 · pp. 7–8

5.2.3 Evaluation Metrics

• Evaluation measures cumulative success by turn and average turns, and introduces hNDCG to account for both success timing and target-item rank.Since each session has one target item, the study uses hDCG; higher SR and hDCG and lower AT indicate better performance.Source: §5.2.3 · p. 7, Table 2 · p. 7

5.3.2 Training Efficiency

• UNICORN trains more stably and reaches better performance in fewer training episodes than strong baselines.SCPR’s curve varies more, while EAR and CRM show little improvement or worsen on the last three datasets with large action spaces.Source: Figure 4 · p. 7, §5.3.2 · p. 8

5.3.3 Comparison at Different Conversation Turns

• UNICORN consistently outperforms baselines across datasets and nearly every conversation turn, especially in the middle turns when many candidate items and attributes remain.SCPR approaches UNICORN in later turns as candidate sets shrink and the task becomes easier.Source: §5.3.3 · p. 8, Figure 5 · p. 8
• Abs Greedy can hit target items early but declines as turns increase, while EAR and CRM resemble it on Yelp* and E-Commerce with large attribute sets.These patterns indicate that EAR and CRM policy learning is limited when the action space is large.Source: §5.3.3 · p. 8

5.4 Ablation Study

• On four of five datasets, removing sequential learning reduced performance more than removing graph-based learning.The authors attribute this to graph knowledge already encoded by pretrained TransE embeddings and the importance of modeling diverse accepted attributes in those datasets.Source: §5.4.1 · p. 8

5.4.1 Model Components.

• Random initialization substantially underperformed pretrained embeddings; TransE further improved on FM, supporting pretraining’s necessity and graph-enhanced RL with limited interaction data [12].The FM embeddings are described as pretrained embeddings from Lei et al. [12].Source: §5.4.2 · pp. 8–9

5.6 Qualitative Analysis

• In sampled E-Commerce conversations, CRM mostly recommended, EAR repeatedly asked unpreferred questions, and SCPR asked some redundant questions; UNICORN comprehensively chose the next action.Figure 6 shows the first five turns of conversations generated by UNICORN, SCPR, EAR, and CRM.Source: Figure 6 · p. 9, §5.6 · p. 9

5.5 Parameter Sensitivity Analysis

• Within 10,000 episodes, UNICORN generally performed best with few candidate items, while selecting only the highest-entropy attribute (Kp=1) risks discarding important attributes.The results also demonstrate the need to prune available actions when UCRPL faces a large action search space.Source: Table 4 · p. 9, §5.5 · p. 9

5.4.2 Pre-training Embeddings.

• Using either preference-only or original entropy-only attribute selection reduced performance, showing both preference and candidate-uncertainty reduction matter.Removing attribute selection hurt large-action applications more than LastFM and Yelp; item selection serves as auxiliary item recall because UNICORN has no pretrained recommendation component.Source: §5.4.3 · p. 9

5.5 Parameter Sensitivity Analysis

• For graph learning, one-hop aggregation was insufficient, but two and three hops performed similarly; one Transformer layer was sufficient for good performance.These findings concern varying the numbers of GCN and Transformer layers.Source: §5.5 · p. 9

6 CONCLUSIONS

• The method significantly outperforms state-of-the-art CRS methods across four benchmark datasets and a real-world E-Commerce application, with scalability and stability.The conclusion reports these outcomes without specifying quantitative results.Source: §6 · p. 9
• The paper formulates when to ask or recommend, what to ask, and which items to recommend as one unified policy learning problem.Source: §6 · p. 9
• It proposes an adaptive reinforcement learning framework based on a dynamic weighted graph for unified conversational recommendation policy learning.Source: §6 · p. 9
• Two action-selection strategies are designed to address the sample efficiency issue.Source: §6 · p. 9