Source-linked AI summary
Grounded Conversation Generation as Guided Traverses in Commonsense Knowledge Graphs
Houyu Zhang, Zhenghao Liu, Chenyan Xiong, Zhiyuan Liu
TL;DR
Conversation models can generate dull, repetitive, or off-topic responses, while human conversations shift across related and multi-hop concepts. ConceptFlow explicitly traverses commonsense knowledge graphs with graph attention to model these flows. On Reddit conversations, it outperforms prior knowledge-aware and GPT-2 systems while using 70% fewer parameters.
Problem
Conversation generation models can produce dull, repetitive, and off-topic responses, while human conversations shift among related and multi-hop concepts.
Method
ConceptFlow grounds conversations in commonsense graphs and uses graph-attention-guided traverses from grounded concepts toward related distant concepts.
Results
ConceptFlow significantly outperforms seq2seq, commonsense-memory, and fine-tuned GPT-2 systems in automatic and human evaluations while using 70% fewer parameters.
Takeaways & Limitations
Explicitly modeling conversation structure with concept flows and graph attention supports more relevant, informative, and less repetitive responses.
Takeaways & Limitations
Using more distant concepts increases graph size; three hops average 3,769 concepts, so the model uses two hops and filters about 200 concepts.
Abstract
from arXiv · showhide
Human conversations naturally evolve around related concepts and scatter to multi-hop concepts. This paper presents a new conversation generation model, ConceptFlow, which leverages commonsense knowledge graphs to explicitly model conversation flows. By grounding conversations to the concept space, ConceptFlow represents the potential conversation flow as traverses in the concept space along commonsense relations. The traverse is guided by graph attentions in the concept graph, moving towards more meaningful directions in the concept space, in order to generate more semantic and informative responses. Experiments on Reddit conversations demonstrate ConceptFlow's effectiveness over previous knowledge-aware conversation models and GPT-2 based models while using 70% fewer parameters, confirming the advantage of explicit modeling conversation structures. All source codes of this work are available at https://github.com/thunlp/ConceptFlow.
1 Introduction
Conversation models can produce dull, repetitive, and off-topic responses, while human dialogue shifts among related and multi-hop concepts. ConceptFlow addresses this by explicitly traversing commonsense concept graphs and achieves stronger response quality with fewer parameters.
- Data-driven conversation models may degenerate into dull, repetitive, off-topic, and useless responses.
- External knowledge can ground conversations, but using only concepts directly present in dialogue does not capture the full potential of knowledge.
- ConceptFlow models conversation flow as traverses through commonsense relations, moving from grounded concepts such as “chat” and “future” toward related concepts such as “talk” and “dream”.
- Graph attention guides traversal toward meaningful relations and distant concepts, supporting more informative and on-topic responses.
- ConceptFlow significantly outperforms seq2seq, commonsense-memory, and fine-tuned GPT-2 systems in automatic and human evaluations while using 70% fewer parameters.
- Analyses find that Reddit conversations align with commonsense paths and that distant concepts improve semantic information while graph attention selects useful concepts.
2 Related Work
Conversation generation research has progressed from seq2seq and pretrained language models to systems that incorporate external textual or graph-based knowledge. Knowledge graphs provide structured entities and relations that can support semantic selection for response generation.
- Seq2seq and pretrained language models are widely used for natural language generation and conversation systems.
- Recent conversation-generation methods incorporate external texts or knowledge graphs to improve response generation.
- Knowledge graphs represent rich semantics through entities and relations, enabling graph attention and knowledge diffusion for selecting response-relevant knowledge.
3 Methodology
ConceptFlow grounds each conversation in a commonsense graph, encodes central and outer concept flows, and combines them with utterance representations during decoding. A hop-selection gate then controls whether generation uses words, central concepts, or two-hop concepts.
- Graph construction: ConceptFlow grounds conversations in a concept graph and traverses commonsense relations toward distant concepts for response generation.
- Base generation model: The model encodes user utterances with an encoder-decoder framework and optimizes generation using cross-entropy loss.
- Graph construction: The graph begins with grounded zero-hop concepts and expands to one-hop and two-hop concepts, forming central and outer graphs.
- Latent concept-flow encoding: Graph neural networks encode the central flow, while relation-focused attention encodes outer flows connecting one-hop concepts to two-hop concepts.
- Context representation: The decoder combines utterance and latent-flow encodings into a context representation for time-step generation.
- Token generation: A gate selects among word generation, central concepts, and outer two-hop concepts before probabilities are computed over the corresponding sets.
- Training: Training and prediction follow conditional language modeling with only ground-truth responses and no additional annotation.
4 Experiment Methodology
The experiments evaluate ConceptFlow on a multi-hop Reddit conversation dataset using relevance, repetitiveness, diversity, and latent concept-flow metrics against Seq2Seq, knowledge-enhanced, and GPT-2 baselines. Implementation constructs a central concept graph and reduces two-hop noise through learned graph-attention selection.
- Dataset: The experiments use a multi-hop Reddit conversation dataset containing 3,384,185 training pairs and 10,000 test pairs.The ConceptNet knowledge graph contains 120,850 triples, 21,471 concepts, and 44 relation types.
- Evaluation Metrics: Response quality is evaluated with PPL, Bleu, Nist, ROUGE, Meteor, Dist-1, Dist-2, and Ent-4 for relevance, repetitiveness, and diversity.These metrics follow prior work and use the implementation from Galley et al. (2018).
- Evaluation Metrics: Precision, Recall, and F1 evaluate whether the learned latent concept flow predicts golden concepts appearing in ground-truth responses.
- Baselines: Knowledge-enhanced baselines store, copy, or attend to grounded concepts but do not explicitly model conversation structures using multi-hop concepts.MemNet stores and reads concepts, CopyNet copies concepts, and CCM uses graph attention for central concepts.
- Implementation Details: Zero-hop concepts are matched from post keywords to ConceptNet, expanded into a central graph, and filtered to the top 100 two-hop concepts using learned graph attention.ConceptFlow (select) is first trained on 10% of random training data to reduce computational cost.
5 Evaluation
ConceptFlow is evaluated through automatic and human response-quality tests, analyses of multi-hop concept selection, hop depth, and attention, plus qualitative case studies. It outperforms baseline systems across reported evaluations, while graph attention and multi-hop concepts support more relevant, informative responses.
- Response Quality: Automatic evaluations show that ConceptFlow produces responses that better match golden responses and balance diversity with novelty.Relevance is measured against golden responses; diversity and novelty use separate metrics for generated text and input-post overlap.
- Response Quality: Human judges score ConceptFlow highest across appropriateness and informativeness, with agreement reaching fair levels measured by Fleiss’ Kappa.Responses are scored from 1 to 4, and reported κ values range from 0.21 to 0.40.
- Effectiveness of Multi-hop Concepts: Graph-attention selection covers more golden concepts than central concepts alone, while reducing the number of selected two-hop concepts relative to adding all golden concepts.The selected concepts also outperform alternative selections in generating ground-truth concepts and token-level perplexity.
- Hop Steps in Concept Graph: Extending the graph from one hop to two and three hops raises golden-concept recall from 39% to 61% and 81%, respectively.Three hops produce 3,769 concepts on average, so the paper uses two hops as a coverage-efficiency balance and filters roughly 200 concepts.
- Case Study: Case studies show ConceptFlow introduces associated multi-hop concepts that make responses more fluent, informative, and diverse than CCM and GPT-2 outputs.One example hops from “music” and “advice” to “sound” and “check.”
6 Conclusion and Future Work
ConceptFlow explicitly models conversation structure as transitions in latent concept space and outperforms previous conversational systems on Reddit conversations. The authors identify future work in combining knowledge with pre-trained language models and introducing more concepts efficiently.
- ConceptFlow models conversation structure as transitions in latent concept space to generate more informative and meaningful responses.
- Experiments on Reddit conversations show advantages over previous conversational systems.
- Future work will combine knowledge with pre-trained language models and introduce more concepts efficiently.
A.1 Supplementary Results for Overall Experiments
Supplementary experiments evaluate relevance and novelty. ConceptFlow achieves stronger relevance and competitive novelty, supporting its ability to generate relevant and informative responses.
- ConceptFlow outperforms other baselines by large margins across supplementary relevance metrics.Concept-PPL combines word and entity distributions, and its value depends on the number of included concepts.
- ConceptFlow has competitive novelty performance against other baselines.Lower novelty scores indicate less repetition between the user post and generated response.
- The supplementary results further support ConceptFlow’s ability to generate relevant and informative responses.
A.2 Supplementary Results for Outer Hop Concepts
Outer-hop selector experiments show that adding and selecting meaningful concepts improves response relevance. ConceptFlow’s selector performs best even without including every golden concept.
- The experiment evaluates four two-hop concept-selection strategies using generated-response quality.
- Adding more concepts improves generated-response quality, while golden-response concepts further improve relevance.Rand outperforms Base on most metrics, and Distract outperforms Rand on all metrics.
- ConceptFlow’s selector significantly outperforms Distract despite excluding some golden concepts.The improvement is attributed to related concepts selected by ConceptFlow.
B Model Details of Central Flow Encoding
Central flow encoding uses a multi-layer graph neural network to propagate concept and utterance representations through a central concept graph. Attention weights combine relation scores with PageRank to control concept flow.
- Central concept encoding: A multi-layer GNN encodes concepts in the central concept graph.Concept embeddings and utterance representations are propagated across layers.
- Central concept encoding: Each concept representation is computed by a feed-forward network over concept and utterance states.The previous-layer neighbor concept and user-utterance representations contribute to the update.
- Graph attention: Relation-specific neighbor semantics are aggregated using attention weights that control concept flow.The relation embedding is combined with neighbor concept information.
- Graph attention: Attention weights depend on relation scores and PageRank scores over neighboring concepts.PageRank controls propagation along paths starting from the initial concept.
- Initialization: Concept embeddings are initialized with pre-trained concept embeddings before graph propagation.
- Evaluation measures: Concept-PPL combines word and concept distributions, but is not comparable across systems using different numbers of concepts.
- Evaluation measures: Novelty is measured against the user utterance, with lower scores indicating better performance.
- Utterance representation: The user-utterance representation is initialized from an utterance hidden state and updated with zero-hop concepts.
C Concept Selection
ConceptFlow selects high-relevance concepts from an exponentially growing, noisy concept space. It ranks two-hop concepts by accumulated attention and combines the top selections with all central concepts to form the concept graph.
- Exponential concept growth introduces noise, motivating selection of high-relevance concepts.
- The selector ranks two-hop concepts using attention scores accumulated across decoder time steps.The score uses decoder output representations and concept embeddings.
- The system reserves the top 100 ranked concepts for the two-hop concept graph.
- All central concepts are retained because they correlate strongly with the conversation topic and remain computationally manageable.
- Central concepts and selected two-hop concepts together construct the concept graph G.