Source-linked AI summary
Topical-Chat: Towards Knowledge-Grounded Open-Domain Conversations
Karthik Gopalakrishnan, Behnam Hedayatnia, Qinlang Chen, Anna Gottardi, Sanjeev Kwatra, Anu Venkatesh, Raefer Gabriel, Dilek Hakkani-Tur
TL;DR
Open-domain conversational AI needs datasets that support knowledge-grounded, broad, and deep conversations without stylized partner roles. The paper introduces Topical-Chat, collects human-human conversations using controlled reading-set symmetry and asymmetry, and benchmarks Transformer-based models through automated and human evaluation. The dataset and evaluations provide a basis for data-driven research in open-domain knowledge-grounded conversational AI.
Problem
Research is limited by scarce knowledge-grounded conversation datasets, while existing datasets often use explicit roles and lack broad, deep topical coverage with transitions.
Method
The paper constructs Topical-Chat from topical reading sets with controlled information symmetry or asymmetry, then trains Transformer-based encoder-decoder models conditioned on conversation history and knowledge.
Results
The paper provides automated and human evaluations of Transformer-based conversational models trained on Topical-Chat for benchmarking knowledge-grounded response generation.
Takeaways & Limitations
Topical-Chat is intended to foster data-driven research in open-domain knowledge-grounded conversational AI.
Takeaways & Limitations
Models used for human evaluation were trained on a subset of the training set.
Abstract
from arXiv · showhide
Building socialbots that can have deep, engaging open-domain conversations with humans is one of the grand challenges of artificial intelligence (AI). To this end, bots need to be able to leverage world knowledge spanning several domains effectively when conversing with humans who have their own world knowledge. Existing knowledge-grounded conversation datasets are primarily stylized with explicit roles for conversation partners. These datasets also do not explore depth or breadth of topical coverage with transitions in conversations. We introduce Topical-Chat, a knowledge-grounded human-human conversation dataset where the underlying knowledge spans 8 broad topics and conversation partners don't have explicitly defined roles, to help further research in open-domain conversational AI. We also train several state-of-the-art encoder-decoder conversational models on Topical-Chat and perform automated and human evaluation for benchmarking.
1. Introduction
Open-domain socialbots must combine world knowledge with conversational history to sustain broad, deep, and smoothly transitioning conversations. Topical-Chat addresses the scarcity of suitable knowledge-grounded datasets and benchmarks Transformer-based response generation with automated and human evaluation.
- Open-domain bots serve as social conversation partners, requiring world knowledge, conversational reasoning, valid responses, and smooth topical transitions.
- The scarcity of knowledge-grounded conversations and associated knowledge sources limits research on learning these conversational skills.
- Topical-Chat contains approximately 11K human-human conversations spanning 8 broad topics, collected from partners given topical reading sets.
- Conversation partners have no explicitly defined roles, and their reading sets may be symmetric or asymmetric to reflect differing prior knowledge.
- Transformer-based encoder-decoder models generate responses from reading sets and conversational history, with automated and human evaluation providing benchmarks.
2. Related Work
Related work includes knowledge-grounded conversation datasets built around movie-focused sources and stylized collection procedures. Sequence-to-sequence systems can produce generic, contextually inappropriate responses, motivating explicit incorporation of world knowledge.
- Earlier datasets included approximately 4K conversations grounded in Wikipedia articles about 30 movies, with article portions shown to partners on a schedule.
- Another movie-focused dataset combined Wikipedia articles, reviews, and web comments about approximately 1K movies and used self-dialogues for collection.
- Sequence-to-sequence response generators condition on previous conversation turns but may produce incoherent, contextually inappropriate, generic, and uninteresting responses.
- Knowledge-grounded approaches address this limitation by incorporating relevant world knowledge into response generation, including through attention-based memory networks.
3. Topical-Chat
Topical-Chat constructs knowledge-grounded conversations from broad topical resources and reading sets with controlled information symmetry or asymmetry. Human partners converse without fixed roles, while the collection process enforces quality and annotates conversational behavior.
- Topical-Chat: Topical-Chat pairs Turkers with topical reading sets and omits explicit conversation roles, using information asymmetry to support teacher-participant dual roles.
- Knowledge Base Creation: The knowledge base is built from entities, facts, and articles, including 300 entities spanning 8 topics.
- Knowledge Base Creation: The knowledge base includes shortened or extractively summarized Wikipedia leads, 8-10 crowdsourced fun facts per entity, and 3088 Washington Post articles.
- Reading Sets Creation: Reading sets are generated in real time from an article and randomized configurations that impose varying degrees of information symmetry or asymmetry.
- Asymmetric Configurations: Config A gives both Turkers the article and shortened Wikipedia leads but different fun facts, creating entity-level fun-fact asymmetry.
- Asymmetric Configurations: Config B gives both Turkers the article and fun facts while differing in whether Wikipedia descriptions are shortened or summarized.
- Conversation Collection: Conversations use experienced Turkers, remain visible alongside reading sets, require at least 20 turns, and include sentiment and other turn-level annotations.
- Conversation Collection: Manual review and automated threshold checks assess acceptability, and validation and test sets are split into frequent and rare entity conditions.
4. Models
The models flatten conversation history and combine it with a selected knowledge sentence before decoding a response. Knowledge selection uses the ground-truth response as an oracle based on TF-IDF similarity.
- Models: The conversation history H_j is formed by concatenating all tokens from the partial conversation C_j.The target x_j+1 is the ground-truth response to be predicted.
- Models: A reading set is tokenized into candidate knowledge sentences, while W_K and W_H truncate knowledge and conversation history respectively.W_K retains tokens from the start of a knowledge sentence, and W_H retains tokens from the end of the history.
- Transformer: The Transformer is trained on (H_j, x_j+1) pairs and decodes a response y from conversation history H during inference.
- Transformer with Knowledge: H_j and the selected sentence k̂ are encoded with a shared Transformer, concatenated, and passed to the Transformer decoder.
- Transformer with Knowledge: Knowledge selection uses x_j+1 in an argmax oracle because selecting knowledge without the ground-truth response remains an open problem.The response and each candidate sentence are represented as TF-IDF vectors for selection.
5. Experiments
The experiments train Transformer models with compact architectural and optimization settings, and compare variants with and without BookCorpus pre-training.
- Experimental Setup: The Transformer uses two layers, two attention heads, a 300-unit feed-forward layer, dropout 0.2, and learned 300-dimensional word embeddings.Position is encoded with one-hot vectors rather than learned positional embeddings.
- Experimental Setup: Training uses batch size 32, stochastic gradient descent, gradient clipping at 0.1, and a learning-rate scheduler with decay 0.5 and patience 3.
- Pre-training: Pre-trained variants use BookCorpus language-model pre-training with byte-pair encoding and a vocabulary size of 37758.Non-pre-trained models do not use BPE and use a vocabulary size of 49957.
6. Results
The paper benchmarks multiple Transformer variants on automated and human evaluations, including frequent and rare test entities. Human ratings indicate generally comprehensible responses and knowledge use by knowledge-ingesting models.
- Evaluation Setup: The evaluation uses model acronyms including TF, w/ p.t., w/ k., and WH = 32 in Tables 3 and 4.
- Dataset Statistics: Table 2 reports statistics for the Topical-Chat conversations and dataset splits.The validation and test sets include frequent entities seen often in training and rare entities seen infrequently or never.
- Human Evaluation: All models were rated as mostly producing comprehensible responses, while knowledge-ingesting models were rated as leveraging knowledge.Agreement was poor for interestingness and willingness to continue, so mean scores were reported for parameters with high agreement.
- Automated Evaluation: Table 3 reports automated metrics on frequent and rare test sets for the evaluated models.
- Human Evaluation: Table 4 reports human evaluation metrics for 150 frequent test snippets.Annotators assessed comprehensibility, topicality, interestingness, knowledge leverage, and willingness to continue the conversation.
- Analysis: Table 5 examines the effect of varying W_H for the Transformer with knowledge on the frequent test set.
7. Conclusion
The paper introduces Topical-Chat as an open-domain, knowledge-grounded dataset and benchmarks Transformer-based response generation with automated and human evaluation. It presents the dataset and models as resources for data-driven research in open-domain conversational AI.
- Conclusion: Topical-Chat contains open-domain conversations without explicit partner roles and covers topical depth, breadth, and transitions.
- Conclusion: The paper trains simple Transformer-based response-generation models and evaluates them with automated metrics and human evaluation.The human-evaluation models were trained on a subset of the training set.
A. Valid / Test Set Creation Strategy
Topical-Chat uses a greedy entity-triplet procedure to construct validation and test splits that distinguish rare from frequent entities.
- Entity frequency is the number of triplets containing an entity, and each triplet receives a score equal to the sum of its constituent entity frequencies.
- The top 10% lowest-scoring triplets are divided into equally sized Vrare and Trare partitions of their corresponding conversations.
- An 80% random selection of triplets defines Dtrain and its corresponding training conversations.
- The remaining conversations are divided into equally sized Vfreq and Tfreq partitions for frequent-entity validation and testing.
B. Topical-Chat: Sample Conversation
The sample conversation’s reading sets use Config A, with asymmetry at the entity level for fun facts. The example centers on Earth, Planet, and Star Wars.
- Config A constructs the reading sets with asymmetry in entity-level fun facts.
- The paper omits these reading sets for brevity.
- The example article is headlined “May the 4th be with Earth, a perfect planet that’s not too Hoth or Tatooine.”Its entities are Earth, Planet, and Star Wars.