Source-linked AI summary
Towards Conversational Recommendation over Multi-Type Dialogs
Zeming Liu, Haifeng Wang, Zheng-Yu Niu, Hua Wu, Wanxiang Che, Ting Liu
TL;DR
The paper addresses how bots can proactively move from non-recommendation dialogs to recommendation in multi-type conversations. It introduces DuRecDial and a multi-goal generation framework, identifying the task and providing a testbed with baseline results for future studies.
Problem
Existing work largely models a single dialog type and assumes users know the conversational goal, leaving proactive transitions across multi-type conversations underexplored.
Method
The paper constructs DuRecDial and develops a multi-goal driven framework with goal planning and goal-guided response generation.
Results
The paper identifies the task, demonstrates DuRecDial’s usability, and provides state-of-the-art model results as baselines for future studies.
Takeaways & Limitations
DuRecDial serves as a testbed for conversational recommendation and related multi-type, knowledge-grounded, domain-transfer, and target-guided dialog tasks.
Takeaways & Limitations
The dataset assumes explicit seeker profiles known to seekers but hidden from recommenders, and the knowledge-selection setup assumes the correct response is conducive to knowledge selection.
Abstract
from arXiv · showhide
We propose a new task of conversational recommendation over multi-type dialogs, where the bots can proactively and naturally lead a conversation from a non-recommendation dialog (e.g., QA) to a recommendation dialog, taking into account user's interests and feedback. To facilitate the study of this task, we create a human-to-human Chinese dialog dataset \emph{DuRecDial} (about 10k dialogs, 156k utterances), which contains multiple sequential dialogs for every pair of a recommendation seeker (user) and a recommender (bot). In each dialog, the recommender proactively leads a multi-type dialog to approach recommendation targets and then makes multiple recommendations with rich interaction behavior. This dataset allows us to systematically investigate different parts of the overall problem, e.g., how to naturally lead a dialog, how to interact with users for recommendation. Finally we establish baseline results on DuRecDial for future studies. Dataset and codes are publicly available at https://github.com/PaddlePaddle/models/tree/develop/PaddleNLP/Research/ACL2020-DuRecDial.
1 Introduction
The paper introduces conversational recommendation over multi-type dialogs, where bots proactively transition from non-recommendation dialogs to recommendation while accounting for user interests. It supports this task with DuRecDial and a multi-goal conversation-generation framework.
- Prior conversational recommender systems mainly model a single dialog type and assume users know the conversational goal from the beginning.
- Multi-type dialogs require bots to transition naturally from chit-chat, QA, or task dialogs toward recommendation, a problem with limited prior work.
- The proposed task emphasizes proactive bot initiative: the recommender plans unknown goal sequences to approach recommendation targets and iterates recommendations with users.
- DuRecDial is a human-to-human Chinese dataset containing natural topic transitions, rich recommendation interactions, explicit seeker profiles, and multiple dialogs per seeker–recommender pair.
- The multi-goal driven conversation generation framework combines goal planning with goal-guided responding for QA, chitchat, recommendation, and task dialogs.
- The paper identifies the task, creates DuRecDial, and proposes a mixed-goal driven dialog policy mechanism as its primary contributions.
2 Related Work
Prior work covers task-oriented and freeform recommendation, knowledge-grounded conversation, and goal-driven open-domain dialog. DuRecDial differs by combining multiple dialog types, multi-domain use cases, explicit goals, rich interactions, and user profiles.
- Datasets for Conversational Recommendation: Existing recommendation datasets include synthetic dialogs, human-to-human recommendation with social chit-chat, clear-goal dialogs, and Dialog↔KG corpora.
- Datasets for Conversational Recommendation: Compared with related datasets, DuRecDial combines multiple dialog types, multi-domain use cases, rich interaction variability, clear goals, and user profiles.
- Datasets for Knowledge Grounded Conversation: Knowledge-grounded dialog datasets vary in document access, domains, and conversational settings, while DuConv supports proactive conversation with an explicit goal.
- Models for Conversational Recommendation: Conversational recommender models generally use either predefined preference slots or learned strategies without task slots; this work adds multi-goal planning across dialog types.
- Goal Driven Open-domain Conversation Generation: Earlier goal-driven open-domain models provide controllability but typically generate dialogs toward a single goal, whereas this work uses a goal sequence beyond in-depth chitchat.
3 Dataset Collection2
DuRecDial is collected through role-based human annotation guided by seeker profiles, knowledge graphs, and task templates. The design supports proactive multi-type recommendation across sequential dialogs, varied user feedback, and annotated goal sequences.
- Task Design: Each dialog pairs a recommendation seeker with a recommender, who proactively guides conversation from non-recommendation scenarios toward an entity recommendation target.Starting scenarios include question answering and social chit-chat; targets are usually aligned with the seeker’s interests.
- Task Design: Seeker profiles encode demographic information and preferences, while the recommender must acquire profile information through dialogue.Profiles are known to seekers, unknown to recommenders, and constrain seeker utterances.
- Task Design: Multiple sequential dialogs model evolving preferences: the first collects profile information, later dialogs recommend based on accumulated preferences, and profiles update after each dialog.Changes in the seeker profile are intended to appear in subsequent dialogs.
- Task Templates: Task templates specify goal sequences and descriptions, with dialog type and topic defining each sub-dialog and rules supporting consistent interests and natural topic transitions.Templates are automatically enumerated and then completed through rules and human annotation.
- Data Collection: The annotation process pairs workers into seeker and recommender roles and uses templates, profiles, and a knowledge graph while requiring every template goal to be tagged.The knowledge graph is built from Baidu Wiki and Douban to support more informative dialogs.
- Data Quality: Three-person evaluation of 200 randomly sampled dialogs yields an average data-quality score of 0.89.Dialogs receive a positive rating when they follow template instructions and are fluent and grammatical.
4 Our Approach
MGCG plans conversational goals and generates goal-appropriate responses using dialogue context, user profiles, knowledge, and explicit goal representations. Its retrieval and generation variants emphasize goals and knowledge to support proactive multi-type conversations.
- Framework overview: MGCG models each dialog using an updated seeker profile, knowledge graph, goal sequence, and goal history to select a current goal and produce a response.The goal-planning module determines where the dialog goes, while the responding module completes the selected goal.
- Goal-planning model: The framework separates goal planning into goal completion estimation and current goal prediction.If the previous goal is incomplete, it remains the current goal; otherwise, the model predicts the next goal's dialog type and topic.
- Retrieval-based response model: The retrieval response model combines context-response, knowledge, goal, knowledge-selection, and matching modules, explicitly emphasizing goals.It encodes candidate responses with context, represents knowledge and goals, fuses information sources, and computes matching probabilities.
- Generation-based response model: The generation model adds an independent goal encoder to a context-and-knowledge generator with context, knowledge, goal, selector, and decoder components.The generator encodes context, conversational goals, and the knowledge graph, then uses goal and knowledge information for response generation.
- Generation-based response model: Knowledge selection learns by minimizing the KL divergence between prior and posterior knowledge distributions, then fuses selected knowledge for decoding.Training uses responses to form the posterior, while testing estimates fused knowledge without ground-truth responses.
- Generation-based response model: The generation objective combines KL-divergence, negative-log-likelihood, and bag-of-words losses, with a trainable weighting parameter.The bag-of-words loss encourages the fused knowledge to remain relevant to the true response.
5 Experiments and Results
Experiments evaluate goal planning, response generation, knowledge use, and human-perceived dialog quality on DuRecDial. Goal and knowledge guidance improves automatic and human evaluation outcomes, while knowledge selection remains important for difficult dialog types.
- Automatic Evaluations: MGCG R and MGCG G outperform S2S across all reported response-generation metrics under matched goal and knowledge settings.MGCG R performs better on Hits@k and DIST-2, whereas MGCG G performs better on knowledge F1.
- Automatic Evaluations: Models using conversational goals and knowledge outperform models without them, confirming their benefits as guidance information.The evaluation includes BLEU, F1, perplexity, DISTINCT, Hits@1, Hits@3, and knowledge precision/recall/F1.
- Human Evaluations: Human evaluations show both MGCG systems outperform S2S, especially on appropriateness, informativeness, goal success rate, and coherence.The retrieval-based model is more fluent, while the generation-based model performs better on the other reported metrics.
- Analysis: Knowledge usage is proportional to goal success across dialog types and methods, while QA and recommendation dialogs are more challenging than chitchat.The analysis identifies stronger knowledge selection for QA and recommendation as an important direction.
6 Conclusion
The paper identifies conversational recommendation over multi-type dialogs and releases DuRecDial as a complex benchmark. It demonstrates dataset usability with baseline results and points to broader future applications.
- Conclusion: DuRecDial contains multiple dialog types and multi-domain use cases for conversational recommendation research.The dataset is presented as a testbed for multi-type dialog modeling and related tasks.
- Conclusion: Baseline results demonstrate the dataset’s usability and provide a basis for future studies.Future applications include knowledge-grounded conversation, domain transfer, target-guided conversation, and multi-type dialog modeling.
1. Dataset collection process
DuRecDial is constructed from seeker profiles, knowledge graphs, goal-sequence templates, and fine-grained recommendation interactions, with human annotation guided by quality controls.
- Profile and Knowledge Collection: Seeker profiles include demographic attributes, domain and entity preferences, and accepted or rejected entity lists that evolve during conversations.Profiles are automatically generated, and later dialogs reflect profile updates.
- Profile and Knowledge Collection: The knowledge graph covers stars, movies, music, news, food, points of interest, and weather.Weather data covers historical observations from 55 cities between July 2017 and August 2019.
- Goal and Interaction Design: Around 20 high-level goal sequences are manually annotated, usually containing 3 to 5 goals across dialog types and domains.Sequences are selected for each seeker according to preferred domains and then enriched to the entity level using seed entities or knowledge-graph similarity.
- Goal and Interaction Design: Fine-grained operations such as rejecting, accepting, questioning, or changing topics are inserted to diversify recommendation interactions.Each interaction operation corresponds to a goal and is randomly inserted into entity-level goal sequences.
- Human Annotation: Two human workers annotate each conversation as bot and user using seeker profiles, knowledge graphs, and task templates.Annotation uses staged worker training, strict standards, and specialist review with re-annotation when requirements are not met.
2. Model Parameter Settings
The paper states that all models are implemented using PaddlePaddle and directs readers to a parameter table for module settings.
- Parameter Settings: All models are implemented using PaddlePaddle.
- Parameter Settings: The parameter settings for all modules are reported in Table 7.
3. Turn-level Human Evaluation Guideline
Turn-level human evaluation assesses response quality through fluency, informativeness, proactivity, and appropriateness, each using ordinal criteria. These criteria distinguish understandable, knowledge-grounded, context-relevant, and proactive responses.
- Fluency: Fluency measures whether a produced response is fluent and understandable.Scores range from 0 for unfluent responses to 2 for fluent, easy-to-understand responses.
- Appropriateness: Appropriateness measures whether a response addresses the context, including relevance, required information, or correctness across dialog types.Scores range from 0 for incorrect or irrelevant responses to 2 when the response otherwise appropriately addresses the context.
- Informativeness: Informativeness measures how fully the response uses knowledge.Scores are 0 when no knowledge is mentioned, 1 when one knowledge triple is mentioned, and 2 when more than one is mentioned.
- Proactivity: Proactivity measures whether the model introduces new knowledge or topics relevant to the conversational context.Irrelevant new topics receive -1, no new topics receive 0, and relevant new topics receive 1.
4. Dialogue-level Human Evaluation Guideline
Dialogue-level human evaluation measures goal completion and coherence, while example conversations illustrate how models use knowledge and pursue conversation goals. The knowledge-aware generator uses more correct knowledge, whereas retrieval and seq2seq systems show limitations in knowledge use or goal completion.
- Goal Completion: Goal Completion measures how well the conversation’s given goals are finished.Score 0 indicates fewer than half the goals are achieved; score 2 requires more than half achieved with full use of knowledge and goal information.
- Coherence: Coherence measures the overall fluency of the dialogue based on irrelevant or contradictory responses.Scores range from 0 when two-thirds of responses are irrelevant or contradictory to 2 when very few are.
- Illustrative Comparison: The knowledge-aware generator uses more correct knowledge for diverse conversation generation.The retrieval-based method produces knowledge-grounded responses but uses relatively little and inappropriate knowledge, while seq2seq fails to complete the goal because knowledge is not fully used.
- Illustrative Comparison: Figure 4 compares model-generated conversations with humans using the conversation goal and related knowledge.The figure uses red for correct context-appropriate knowledge, blue for inappropriate knowledge, and purple for correct knowledge with an inappropriate response.