Source-linked AI summary
Estimation-Action-Reflection: Towards Deep Interaction Between Conversational and Recommender Systems
Wenqiang Lei, Xiangnan He, Yisong Miao, Qingyun Wu, Richang Hong, Min-Yen Kan, Tat-Seng Chua
TL;DR
Static recommenders may miss users’ changing online preferences, motivating a unified CRS framework for asking questions, timing recommendations, and adapting to feedback. EAR addresses these problems through estimation, action selection, and reflection, and experiments report improved CRS performance over prior methods.
Problem
Static recommender systems rely on offline data that may not match users’ diverse and time-varying online preferences, leaving CRS interaction problems unified inadequately.
Method
EAR estimates item and attribute preferences, uses a policy to ask attributes or recommend items, and updates the recommender after rejected recommendations.
Results
EAR significantly outperforms other methods in success rate and average turns across the evaluated conversational recommendation settings.
Takeaways & Limitations
The results support closely integrating conversation and recommendation components in multi-round conversational recommendation.
Takeaways & Limitations
The policy network is not updated online after recommendation rejection, which the paper leaves for future work.
Abstract
from arXiv · showhide
Recommender systems are embracing conversational technologies to obtain user preferences dynamically, and to overcome inherent limitations of their static models. A successful Conversational Recommender System (CRS) requires proper handling of interactions between conversation and recommendation. We argue that three fundamental problems need to be solved: 1) what questions to ask regarding item attributes, 2) when to recommend items, and 3) how to adapt to the users' online feedback. To the best of our knowledge, there lacks a unified framework that addresses these problems. In this work, we fill this missing interaction framework gap by proposing a new CRS framework named Estimation-Action-Reflection, or EAR, which consists of three stages to better converse with users. (1) Estimation, which builds predictive models to estimate user preference on both items and item attributes; (2) Action, which learns a dialogue policy to determine whether to ask attributes or recommend items, based on Estimation stage and conversation history; and (3) Reflection, which updates the recommender model when a user rejects the recommendations made by the Action stage. We present two conversation scenarios on binary and enumerated questions, and conduct extensive experiments on two datasets from Yelp and LastFM, for each scenario, respectively. Our experiments demonstrate significant improvements over the state-of-the-art method CRM [32], corresponding to fewer conversation turns and a higher level of recommendation hits.
1 INTRODUCTION
The paper frames conversational recommendation as a multi-round interaction between conversation and recommendation, introducing EAR to estimate preferences, choose actions, and adapt to feedback.
- Static recommender systems rely on offline historical data, which may not match users’ diverse and time-varying online preferences.
- Multi-round CRS interactions alternate between asking about attributes and recommending items while minimizing conversation turns.
- The central interaction problems are which attributes to ask, when to recommend, and how to use users’ online feedback.
- EAR estimates preferences for items and attributes, learns dialogue actions, and updates the recommender after rejected recommendations.
- The paper considers binary and enumerated attribute questions, while avoiding open questions that do not constrain responses.
- EAR is evaluated with simulated CRS datasets and is reported to outperform state-of-the-art CRS methods.
2 MULTI-ROUND CONVERSATIONAL RECOMMENDATION SCENARIO
The paper defines conversational recommendation as a multi-round session in which the CRS repeatedly asks about attributes or recommends items until success or departure.
- A round is one trial of recommendation, and the multi-round setting permits repeated asking and recommending until success or the user leaves.
- Each item has attributes, and the session begins with the user specifying a preferred attribute used to filter candidate items.
- At every turn, the CRS chooses either recommendation or an attribute question.
- A positive response to a recommendation ends the session, whereas rejection moves the interaction to the next round.
3 PROPOSED METHODS
EAR coordinates the recommender and conversation components through estimation, action selection, and adaptation to conversational feedback.
- The recommender ranks candidate items and attributes to support the conversation component’s action decision.
- The conversation component chooses whether to ask an attribute or recommend items using ranked candidates, ranked attributes, and dialogue history.
3.1 Estimation
The Estimation stage predicts item and attribute preferences using shared factorization-machine representations and trains item ranking with both general and candidate-specific negatives.
- The recommender accumulates preferred attributes during conversation and predicts both preferred items and preferred attributes.
- The factorization-machine item score combines general user–item interest with affinity between the item and the user’s preferred attributes.
- The model uses BPR training, ranking interacted items above non-interacted items.
- Attribute-aware BPR adds negative samples from current candidate items satisfying the user’s partially known preferences.
- The combined losses learn general preference and specific preference among dynamically changing conversation candidates.
- Adding the candidate-specific loss is described as critical for ranking current candidates, unlike CRM.
- Item and attribute prediction are jointly trained because their parameters are shared and may provide mutual benefits.
3.2 Action
The Action stage uses reinforcement learning to choose whether to ask about attributes or recommend items, using a state vector that combines recommender signals and dialogue history. Its policy optimizes successful recommendations in fewer turns.
- 3.2 Action: The Action stage applies reinforcement learning to multi-round decisions about when to ask attributes or recommend items.The policy aims to achieve successful recommendation in a shorter number of turns.
- 3.2 Action: The state vector bridges the conversational and recommender components by concatenating four vectors encoding distinct information sources.These sources include recommender estimates and dialogue-history signals.
- 3.2 Action: Attribute entropy estimates which questions may most reduce the candidate-item space and help find desirable items in fewer turns.The entropy vector has one dimension for each attribute, with larger entropy indicating a potentially more useful question.
- 3.2 Action: A two-layer multilayer perceptron maps the state vector to a softmax probability distribution over all attribute-asking actions and recommendation.The policy network is optimized with the standard policy-gradient method.
- 3.2 Action: The policy-gradient objective uses discounted future rewards, with the return accumulating rewards from the current turn through the final turn.The discount factor weights future rewards relative to immediate rewards.
3.3 Reflection
The Reflection stage updates the recommender after rejected recommendations by treating rejected items as negative examples. It does not update the dialogue policy online because doing so is considered difficult for the reinforcement-learning agent.
- 3.3 Reflection: Reflection is triggered by rejected recommendations and updates the recommender model using the rejected items as negative samples.This exploits explicit online dislike feedback that is absent from traditional static recommender training.
- 3.3 Reflection: The online update constructs training triplets by pairing historically interacted positive items with rejected recommendation items.Because the true positive item is unavailable online, historically interacted items serve as positives.
- 3.3 Reflection: The Reflection stage optimizes the BPR loss with batch gradient descent, converging empirically in 3-5 epochs for online use.The training examples are collected in D4 from historical positives and rejected items.
- 3.3 Reflection: The policy network is not updated from rejection feedback because online reinforcement-learning updates are considered highly difficult.Updating the conversational component is left for future work.
4 EXPERIMENTS
The experiments evaluate EAR’s interaction framework through overall comparisons and component ablations on Yelp and LastFM, covering binary and enumerated attribute-question settings. Evaluation measures recommendation success and conversation efficiency against several baselines.
- 4 EXPERIMENTS: The experiments investigate overall performance and the contributions of estimation, action-state design, and online reflection through four research questions.The questions separately examine overall comparison, estimation components, state-vector appropriateness, and online model updating.
- 4 EXPERIMENTS: Experiments use Yelp business recommendations and LastFM music-artist recommendations, with interactions split 7:2:1 for training, validation, and testing.Users with fewer than 10 reviews are pruned to reduce sparsity.
- 4 EXPERIMENTS: The datasets contain 590 Yelp attributes and 33 LastFM attributes, supporting enumerated questions on Yelp and binary questions on LastFM.Yelp uses a two-level attribute taxonomy, and the policy outputs 29 parent attributes plus recommendation.
- 4 EXPERIMENTS: A user simulator generates training and evaluation conversations from observed user-item interactions, treating each interacted item and its attributes as the session target and oracle.Sessions use a maximum of 15 turns and recommendation lists of length 10.
- 4 EXPERIMENTS: EAR is compared with Max Entropy, Abs Greedy, and CRM under adapted multi-round training and evaluation settings.CRM is described as a state-of-the-art CRS that integrates conversational and recommender components but has limited interaction between them.
- 4 EXPERIMENTS: Success rate by turn and average turns measure recommendation effectiveness and conversation efficiency, while AUC evaluates the offline recommender model.One-sample paired t-tests assess statistical significance.
4.2 Performance Comparison (RQ1)
EAR significantly outperforms competing methods in recommendation success, while attribute-based methods generally become more effective than recommendation-only behavior as conversations continue. Its learned strategy differs across Yelp’s enumerated and LastFM’s binary settings.
- 4.2 Performance Comparison (RQ1): EAR significantly outperforms other methods in success rate across conversation turns on Yelp and LastFM.Figure 2 reports Success Rate* relative to CRM, and Table 2 reports final success rate and average turns.
- 4.2 Performance Comparison (RQ1): Attribute-based methods initially have nearly zero success rate before turn 2 because they ask questions at the beginning of conversations.Abs Greedy recommends items every turn and therefore behaves differently early in the session.
- 4.2 Performance Comparison (RQ1): As conversations continue, Abs Greedy falls behind attribute-based methods, demonstrating the value of asking attributes in this recommendation setting.Its performance is notably weaker on Yelp, where enumerated responses can contain multiple fine-grained attributes.
- 4.2 Performance Comparison (RQ1): CRM generally underperforms EAR, especially in the larger action space of nearly 30 attributes compared with 5 in CRM’s original setting.The paper attributes this gap partly to CRM’s state vector being less able to support sophisticated asking and recommendation strategies.
- 4.2 Performance Comparison (RQ1): On Yelp, EAR’s gain over CRM enlarges in turns 1–3, shrinks in turns 4–6, and widens again afterward, whereas LastFM shows a steadily increasing gain.The paper relates this pattern to different learned strategies for enumerated versus binary questions.
4.3 Effectiveness of Estimation Designs (RQ2)
The estimation stage combines attribute-aware ranking with multi-task training to improve item and attribute prediction. State-vector ablations show that useful decision signals differ between binary and enumerated question settings.
- Estimation designs: Attribute-aware BPR improves item ranking but can reduce attribute prediction, including LastFM AUC falling to 0.629 from FM’s 0.727.The ranking-oriented loss specifically fits item ordering in the candidate list.
- Estimation designs: FM+A+MT significantly enhances both item recommendation and attribute prediction, with statistically significant improvements over FM and FM+A at p < 0.01.Table 3 evaluates FM, FM+A, and FM+A+MT on offline AUC for both tasks.
- State-vector ablation: Question type determines the most important state signal: sent for LastFM’s binary questions and slen for Yelp’s enumerated questions.Entropy information is crucial on LastFM, whereas candidate-list length is most important on Yelp.
- State-vector ablation: Ablating sent on LastFM reaches 0.051 in SR@5 but substantially harms future SR because the system recommends before gathering sufficient attribute evidence.For Yelp, slen helps decide when to recommend as the candidate list shrinks more sharply.
- State-vector ablation: Attribute preference and conversation history contribute positively, but their effects vary by dataset and metric.On Yelp, spre strongly affects SR@5 and SR@10 but has no significant impact on SR@15.
4.5 Investigation on Reflection (RQ4)
Reflection updates the recommender after rejected recommendations, but its benefit varies across datasets. On Yelp, updates can harm ranking when the offline model already performs well.
- Reflection ablation: Online updating helps substantially on LastFM but has only a very minor effect on Yelp.Table 5 compares EAR with a variant that removes the online-update module.
- Bad updates: Yelp updates are often harmful when the offline model ranks the ground-truth item high but outside the top 10.Such updates are highly likely to pull the ground-truth item downward.
- Bad updates: Approximately 3.5% of bad updates come from users with an offline AUC of 0.9 on Yelp.The paper reports a clear positive correlation between bad-update percentage and offline AUC.
- Cross-dataset interpretation: Online update works better on LastFM than Yelp alongside offline AUC values of 0.742 and 0.870, respectively.The paper links Yelp’s stronger offline ranking to a greater likelihood of harmful updates.
5 RELATED WORK
Prior recommendation methods rely mainly on offline historical interactions and therefore struggle to capture changing preferences. Conversational systems address this limitation by eliciting dynamic feedback, but prior CRS work often uses simplified settings.
- Static recommendation: Static recommendation methods estimate user–item affinity from historical interactions using approaches such as Matrix Factorization and Factorization Machines.Neural FM and DeepFM extend FM with deep neural networks.
- Static recommendation: Static methods cannot capture users’ dynamic preferences.This limitation motivates online recommendation approaches that adapt results using users’ online actions.
- Conversational recommendation: Conversational recommender systems capture dynamic feedback by interacting with users through natural language.The field includes varied settings and problem formulations.
- Conversational recommendation: Prior CRS work generally considers conversational recommendation under simplified settings.The cited examples illustrate that earlier systems restrict the conversational recommendation setup in various ways.
6 CONCLUSION AND FUTURE WORK
The paper frames EAR as a unified framework for accurate conversational recommendation in fewer turns by coordinating conversational and recommender components. It reports experimental validation while identifying online feedback, policy refinement, exploration–exploitation, and real-user deployment as future directions.
- EAR unifies the problems of what to ask, when to recommend, and how to adapt to feedback within a conversational recommendation framework.
- Experiments on two datasets justify EAR’s effectiveness and provide insights into conversational strategy and online updates.
- The authors identify incorporating user feedback as a loose end requiring further investigation.
- Future work includes refreshing the policy network, studying explore–exploit balance, and deploying the system with real users.