Source-linked AI summary
Proactive Human-Machine Conversation with Explicit Conversation Goals
Wenquan Wu, Zhen Guo, Xiangyang Zhou, Hua Wu, Xiyuan Zhang, Rongzhong Lian, Haifeng Wang
TL;DR
The paper addresses the limited proactivity of dialogue systems that typically respond passively and do not explicitly exploit associated knowledge. It introduces DuConv and knowledge-aware dialogue models that plan over a knowledge graph to guide goal-directed topic shifts. Results show greater knowledge use and more diverse conversations, although 15% to 20% of goals remain unachieved because the models lack an explicit multi-turn policy.
Problem
Dialogue systems often rely only on utterances and responses, while existing knowledge-aware systems generally answer questions rather than proactively leading conversations.
Method
The paper builds DuConv, a crowdsourced leader-follower dataset, and develops dialogue models that plan strategies over a knowledge graph using explicit conversation goals.
Results
Knowledge-using models outperform models without knowledge, while proposed generation methods outperform Seq2Seq and retrieval models in human-rated informativeness, proactivity, goal completion, and coherence.
Takeaways & Limitations
Planning over a knowledge graph enables models to use more related knowledge and generate more diverse multi-turn conversations; the dataset and models are publicly available.
Takeaways & Limitations
15% to 20% of conversation goals remain unachieved because the models lack an explicit multi-turn policy mechanism controlling the whole conversation flow.
Abstract
from arXiv · showhide
Though great progress has been made for human-machine conversation, current dialogue system is still in its infancy: it usually converses passively and utters words more as a matter of response, rather than on its own initiatives. In this paper, we take a radical step towards building a human-like conversational agent: endowing it with the ability of proactively leading the conversation (introducing a new topic or maintaining the current topic). To facilitate the development of such conversation systems, we create a new dataset named DuConv where one acts as a conversation leader and the other acts as the follower. The leader is provided with a knowledge graph and asked to sequentially change the discussion topics, following the given conversation goal, and meanwhile keep the dialogue as natural and engaging as possible. DuConv enables a very challenging task as the model needs to both understand dialogue and plan over the given knowledge graph. We establish baseline results on this dataset (about 270K utterances and 30k dialogues) using several state-of-the-art models. Experimental results show that dialogue models that plan over the knowledge graph can make full use of related knowledge to generate more diverse multi-turn conversations. The baseline systems along with the dataset are publicly available
1 Introduction
The paper targets dialogue systems that passively respond without explicitly exploiting related knowledge, proposing proactive conversation guided by explicit goals over a knowledge graph. It introduces DuConv and knowledge-aware models for leading natural, engaging topic shifts.
- Motivation: Existing dialogue systems often rely only on utterances and responses, producing uninformative or inappropriate replies despite knowledge-grounded alternatives.Prior knowledge-aware systems generally answer questions rather than proactively asking questions or leading conversations.
- Proposed task: The paper formulates proactive conversation as planning over a knowledge graph to follow an explicit path from a starting point through topic a to topic b.Each topic represents an entity in the knowledge graph.
- Dataset: DuConv contains around 30k multi-turn conversations created by paired crowdsourced workers assigned leader and follower roles.The leader receives the goal and related knowledge, while the follower responds without that information.
- Dataset: The dataset requires leaders to sequentially change topics according to the goal while keeping dialogue natural and engaging.This combines conversational understanding with knowledge-graph planning.
- Contributions: Knowledge-aware proactive dialogue models are proposed, and experiments report proactive goal completion and greater use of provided knowledge.The contributions also include a released large-scale dataset and detailed model analysis.
2 Related Work
Related work spans proactive conversation and knowledge-grounded conversation. Prior proactive systems commonly target specific task-oriented or social-dialogue scenarios, while knowledge-grounded work uses external knowledge for more engaging responses and releases supporting datasets.
- Scope: The paper positions its work at the intersection of proactive conversation and knowledge-grounded conversation.The related-work discussion identifies these as its two major research topics.
- Proactive Conversation: Proactive conversation research aims to give dialogue systems the ability to lead interactions.Existing examples include actively questioning or clarifying slots in task-oriented dialogue and asking questions in social bots.
- Knowledge Grounded Conversation: Knowledge-grounded conversation research exploits external knowledge to generate engaging, meaningful, or personalized responses.This research line includes both improved conversation models and released knowledge-grounded datasets.
3 DuConv
DuConv is built through knowledge crawling, graph construction, goal assignment, and crowdsourced conversation collection. Its movie-focused graph combines structured and unstructured knowledge, while dialogues use paired leaders and followers to realize two-step topic shifts.
- Knowledge Crawling: DuConv limits dialogue topics to movies and film stars and crawls related information from MTime.com.The collection includes structured facts and unstructured comments and synopses, with dirty or offensive information filtered.
- Knowledge Graph Construction: The knowledge graph contains SPO triplets whose objects may be factoid facts or non-factoid sentences such as comments and synopses.The graph therefore combines structured relations with textual knowledge.
- Knowledge Graph Construction: Direct triplets encode one-step relations, whereas associated triplets connect entities sharing a predicate and object as two-step relations.An associated triplet is formed by combining two original triplets.
- Conversation Goal Assignment: About 30k sampled knowledge paths serve as conversation goals, shifting naturally from a start state to topic a and then topic b.The two topics are sampled linked entities in the knowledge graph.
- Conversation Crowdsourcing: For each goal, two crowdsourced workers act as leader and follower; the leader receives the goal and graph knowledge, while the follower sees only dialogue history.The dialogue continues until the leader achieves the assigned goal.
4 Methods
The paper develops retrieval-based and generation-based dialogue models that incorporate dialogue context, explicit goals, and related knowledge to support proactive conversation. Both architectures represent and select knowledge, while the generator additionally plans knowledge selection and trains with multiple objectives.
- Retrieval-based Model: The retrieval-based system retrieves candidate responses and selects the best one using dialogue context, goal, and related knowledge.Its ranker includes context-response encoding, knowledge representation, knowledge reasoning, and matching modules.
- Retrieval-based Model: The response ranker encodes context and candidates with BERT-style self-attention and encodes each knowledge item with a bidirectional GRU.The resulting context-response representation attends to knowledge vectors before matching.
- Retrieval-based Model: The retrieval model fuses knowledge-side and dialogue-side representations, then computes the probability that a candidate is appropriate for the context, goal, and knowledge.The dialogue goal is treated as part of the knowledge used in the conversation.
- Generation-based Model: The generation-based model extends seq2seq with utterance encoding, knowledge encoding, knowledge management, and decoding components.It uses an external knowledge-selection mechanism to support knowledge-driven response generation.
- Generation-based Model: The generator uses prior and posterior knowledge distributions, aligns them with KL divergence during training, and estimates fused knowledge from the prior distribution at test time.The decoder uses a hierarchical gated fusion unit with external knowledge gates.
- Generation-based Model: The generator combines KL-divergence, negative-log-likelihood, and bag-of-words losses.The bag-of-words loss enforces relevance between fused knowledge and the true response, while NLL measures generated-response error.
5 Experiments
The experiments evaluate DuConv models with automatic and human metrics, comparing knowledge use, normalization, retrieval, generation, and baselines. Knowledge-aware generation performs strongly on human measures, but 15%–20% of goals remain incomplete without explicit multi-turn policy control.
- Evaluation Settings: Automatic evaluation uses BLEU, PPL, F1, DISTINCT1/2, and Hits@K to measure fluency, relevance, diversity, and retrieval performance.Models select the best response from ten candidates, including one human response and nine randomly sampled responses.
- Evaluation Settings: Human evaluation measures turn-level fluency, coherence, informativeness, and proactivity, plus dialogue-level goal completion and coherence.Three annotators evaluate responses or complete generated dialogues against conversation goals and related knowledge.
- Compared Models: The comparison includes vanilla Seq2Seq, retrieval-based, generation-based, normalized, and knowledge-ablated models.The ablation replaces all knowledge triplets with “UNK, UNK, UNK”; normalized models replace the two path topics with “topic a” and “topic b.”
- Results: Retrieval-based models perform better on Hits@K but worse on F1 and BLEU, while generation-based models perform better in human evaluation.The authors attribute these differences to different optimization objectives and the retrieval model’s limited candidate set.
- Results: Knowledge use improves performance, and topic normalization significantly improves all models through better generalization over knowledge.Human evaluations cover normalized models because they achieve better performance; annotator agreement ranges from 0.37 to 0.86.
- Results: Proposed generation methods outperform Seq2Seq and retrieval models in informativeness, proactivity, goal completion, and dialogue coherence.The generation model uses more knowledge to achieve score “2” goal completion, whereas Seq2Seq more often receives score “1” and produces duller conversations.
- Limitations: 15% to 20% of conversation goals remain unachieved because the models lack an explicit multi-turn policy mechanism.The paper leaves controlling the whole conversation flow for future research.
6 Case Study
The case study compares knowledge-aware generation, retrieval, and Seq2Seq conversations under a goal and related knowledge. The generator selects more appropriate knowledge for diverse dialogue, whereas retrieval often uses wrong knowledge and Seq2Seq remains generic.
- Case Study: The case study presents model conversations with humans given a conversation goal and related knowledge.The examples illustrate how each model uses knowledge while conducting the dialogue.
- Knowledge-Aware Generation: The knowledge-aware generator chooses more appropriate and more knowledge for diverse conversation generation.Its broader knowledge use supports conversations that are more varied than those produced by the other systems.
- Retrieval-Based Method: The retrieval-based method produces knowledge-grounded responses, but the used knowledge is often wrong.The case study contrasts grounding with correctness of the selected knowledge.
- Seq2Seq Baseline: Seq2Seq can smoothly achieve the knowledge goal but relies on generic safe responses, making conversations less diverse and sometimes dull.The passage links this behavior to mentioning much less knowledge than the knowledge-aware generator.
7 Conclusion
The paper introduces proactive conversation over a knowledge graph and releases DuConv as a benchmark for this task. Its experiments indicate that planning over the graph enables fuller knowledge use and more diverse conversations.
- Conclusion: The paper builds a conversational agent capable of proactively leading conversations.The conclusion frames proactive leadership as the central agent capability.
- Conclusion: DuConv contains dialogues created by two crowdsourced workers acting as leader and follower.The leader receives a knowledge graph and sequentially changes topics according to a conversation goal while maintaining natural, engaging dialogue.
- Conclusion: DuConv requires agents to plan topic changes over a knowledge graph while following an explicit conversation goal.The example conversation includes a path from McDull: Rise of the Rice Cooker to Bo Peng and a rating of 6.9.
- Conclusion: Knowledge-graph planning helps dialogue models use related knowledge more fully to generate more diverse conversations.The dataset and models are publicly available as benchmarks for future knowledge-driven proactive dialogue research.
A. Turn-level Human Evaluation Guideline
The turn-level guideline defines four response qualities: fluency, coherence, informativeness, and proactivity. Each quality is scored using criteria tied to understandability, contextual relevance, knowledge use, or topic introduction.
- Fluency: Fluency measures whether a response is fluent, with scores from 0 for difficult-to-understand text to 2 for fluent text.Score 1 denotes errors that do not prevent understanding.
- Coherence: Coherence measures whether a response addresses the context, penalizing irrelevant knowledge or failure to answer the user’s question.Scores range from 0 for contradiction or irrelevance to 2 for an otherwise suitable response.
- Informativeness: Informativeness measures knowledge use: score 0 mentions no knowledge, score 1 mentions one triplet, and score 2 mentions more than one.The scale directly evaluates how fully the response exploits the provided knowledge.
- Proactivity: Proactivity measures whether new knowledge or topics relevant to the context are introduced.Scores are -1 for irrelevant new topics, 0 for none, and 1 for relevant new topics.
B. Dialogue-level Human Evaluation Guideline
The guideline evaluates dialogue-level goal completion and overall coherence using ordinal scores based on topic coverage, knowledge-triplet use, relevance, and logical consistency.
- Goal Completion measures how well the given conversation goal is finished.
- Goal Completion: A score of 0 means neither topic a nor topic b is mentioned.
- Goal Completion: A score of 1 indicates one topic is mentioned, but the dialogue is boring and uses fewer than 3 different knowledge triplets.
- Goal Completion: A score of 2 requires both topics to be mentioned and more than 2 different knowledge triplets to be used.
- Coherence: Coherence measures dialogue fluency on a 0–3 scale according to the number of irrelevant or logically contradictory responses.The scale ranges from over 2 problematic responses for score 0 to none for score 3.