Source-linked AI summary
Knowledge-Grounded Dialogue Generation with Pre-trained Language Models
Xueliang Zhao, Wei Wu, Can Xu, Chongyang Tao, Dongyan Zhao, Rui Yan
TL;DR
Knowledge-grounded dialogue generation must combine specific external knowledge with pre-trained language models despite redundant inputs and fixed capacity. The paper adds a knowledge selection module and jointly optimizes it with response generation using unlabeled dialogues, significantly outperforming state-of-the-art methods on two benchmarks.
Problem
Pre-trained language models have limited input capacity, while retrieved knowledge can be redundant and too long, and ground-truth knowledge labels are unavailable.
Method
The paper equips a pre-trained response generator with a BERT-based knowledge selection module and jointly optimizes selection and generation using unlabeled dialogues.
Results
The model significantly outperforms state-of-the-art methods on the Wizard of Wikipedia and CMU Document Grounded Conversations benchmarks.
Takeaways & Limitations
Knowledge selection can let pre-trained response generators use relevant information from redundant external knowledge under capacity constraints.
Takeaways & Limitations
The approach assumes that knowledge is retrieved from documents and relies on unlabeled dialogues rather than ground-truth knowledge labels.
Abstract
from arXiv · showhide
We study knowledge-grounded dialogue generation with pre-trained language models. To leverage the redundant external knowledge under capacity constraint, we propose equipping response generation defined by a pre-trained language model with a knowledge selection module, and an unsupervised approach to jointly optimizing knowledge selection and response generation with unlabeled dialogues. Empirical results on two benchmarks indicate that our model can significantly outperform state-of-the-art methods in both automatic evaluation and human judgment.
1 Introduction
Open-domain dialogue systems often produce bland responses, while knowledge grounding introduces long, redundant inputs that exceed pre-trained language-model capacity. The paper addresses this tension with unsupervised knowledge selection jointly optimized with response generation, achieving strong benchmark results.
- Generative open-domain dialogue models are prone to generic, bland responses, especially when conversations require specific knowledge.
- Pre-trained language models provide stronger contextual generation but may still produce bland or inappropriate responses when specific knowledge is required.
- Knowledge-grounded dialogue datasets contain redundant retrieved knowledge, creating a conflict between rich inputs and fixed model token capacity.GPT-2, for example, has a 1024-token maximum, while the described knowledge averages 1625.6 tokens.
- The proposed knowledge selection module retains context-relevant information while slimming redundant knowledge to fit the pre-trained generator's capacity.
- An unsupervised approach jointly learns knowledge selection and response generation from unlabeled dialogues using pseudo ground-truth, reinforcement learning, and curriculum learning.
- The model significantly outperforms state-of-the-art methods on Wizard of Wikipedia and CMU Document Grounded Conversations.
2 Related Work
Prior work improves open-domain dialogue through diverse response modeling, conversation-context structure, and large-scale pre-training. Knowledge-grounded dialogue addresses specificity, while this paper investigates using existing pre-trained models when inputs exceed their capacity.
- Early open-domain dialogue research extended encoder-decoder models to improve response diversity and model conversation-context structure.
- Large pre-trained language models have produced strong improvements across natural language understanding and generation benchmarks.
- This work differs from low-resource pre-training research by investigating how to use existing pre-trained language models when input exceeds model capacity.
3 Preliminary
The task learns a GPT-2 response model from dialogue contexts, documents, and responses, while defining an input transformation that fits context and knowledge within model limits. Because knowledge labels are unavailable and truncation can discard relevant information, the paper motivates learning this transformation.
- 3.1 Problem Formalization: Each training example contains a dialogue context U, a relevant document D, and a response r grounded in D.
- 3.1 Problem Formalization: The objective is to learn P(r|U, D; θ) so responses can be generated for new contexts paired with documents.
- 3.2 Pre-trained Language Models: GPT-2 generation conditions each response token on a transformed context-document input g(U, D) and preceding response tokens.
- 3.2 Pre-trained Language Models: The transformation g(U, D) must tailor the concatenated context and document to satisfy GPT-2's length constraint.
- 3.2 Pre-trained Language Models: Without knowledge labels, simple truncation may remove relevant information and introduce noise, motivating a learned distillation function g(U, D).
4 Approach
The approach selects relevant external-knowledge sentences with a context-aware, sequential module and jointly trains selection and GPT-2 response generation without human knowledge labels. Pseudo ground-truth initialization, reinforcement learning, and curriculum training coordinate the two components.
- 4.1 Context-Aware Knowledge Encoder: The model uses a context-aware knowledge encoder built on BERT to represent each candidate knowledge sentence alongside the dialogue context.Each encoded candidate is represented by its [CLS] vector, producing E = (e1, ..., em).
- 4.2 Sequential Knowledge Selector: The sequential selector constructs D′ by adding candidate sentences step by step, allowing the selected-set size to vary with the context.Selection terminates when a special embedding is chosen or the upper bound Tmax is reached.
- 4.2 Sequential Knowledge Selector: The selected sentences define g(U, D) by concatenating U with D′ for GPT-2 response generation.This selection is intended to distill useful knowledge from redundant candidates rather than relying on simple truncation.
- 4.3 Learning Method: Pseudo ground-truth is constructed from response–knowledge relevance signals to warm up knowledge selection and GPT-2 fine-tuning before joint optimization.The resulting knowledge-selection data DK contains dialogues, candidate knowledge, and pseudo ground-truth subsets.
- 4.3 Learning Method: A reinforcement step continues training the selector with policy gradients, using GPT-2 responses to reward selections according to a target metric.The baseline b reduces gradient-estimation variance, and selections outperforming it receive higher conditional likelihood.
- 4.3 Learning Method: Curriculum learning gradually replaces pseudo ground-truth with model-selected knowledge while fine-tuning GPT-2 on unlabeled dialogues.The training algorithm initializes both modules with maximum likelihood estimation and exponentially shrinks the curriculum rate p.
5 Experiments
Experiments evaluate KnowledGPT on Wizard of Wikipedia and CMU DoG using automatic and human judgments, comparing it with established baselines and pre-trained-model heuristics. Results show strong benchmark performance, benefits from joint learning strategies, and a trade-off controlled by Tmax.
- Experimental Setup: Experiments use Wizard of Wikipedia and CMU Document Grounded Conversations, with perplexity, BOW Embedding, and unigram F1 as automatic metrics.Human evaluation samples 300 examples from Wizard Test Seen, Test Unseen, and CMU DoG, with three native-speaker annotators judging response quality.
- Baselines: KnowledGPT is compared with Transformer Memory Network, Incremental Transformer with Deliberation Decoder, Sequential Knowledge Transformer, DRD, and GPT-2trunc.The comparison also includes pre-trained models used with heuristic knowledge handling and records whether models use human labels.
- Automatic Evaluation: KnowledGPT achieves new state-of-the-art performance on most metrics in both datasets.GPT-2trunc is worse when knowledge is cut or when redundant and irrelevant knowledge remains in the input; ground-truth knowledge is cut in 53% of Wizard test examples.
- Human Evaluation: KnowledGPT is superior on context coherence and knowledge relevance, while the three evaluated models are comparable on fluency.All Fleiss’ kappa values are at least 0.6, indicating substantial annotator agreement.
- Ablation Study: Removing pseudo-ground-truth warming up or joint optimization causes performance drops, while reinforcement and curriculum steps are useful.The pseudo-ground-truth step is especially important on Wizard, where knowledge correlates strongly with human responses; CMU DoG workers refer to external knowledge less often.
- Discussions: Increasing Tmax lowers perplexity by allowing more chances to include the ground-truth candidate, but excessive Tmax introduces noise and causes F1 to decline.Thus, the termination setting balances access to relevant knowledge against noisy generation.
6 Conclusions
The paper applies large-scale pre-trained language models to knowledge-grounded dialogue generation by combining knowledge selection with response generation. An unsupervised joint optimization approach achieves significant improvements over state-of-the-art methods on two benchmarks.
- The model applies large-scale pre-trained language models to knowledge-grounded dialogue generation.
- It devises a knowledge selection module and jointly optimizes knowledge selection with response generation without supervision.
- Evaluation on two benchmarks indicates significant outperformance of state-of-the-art methods.
A Details of Datasets
This section reports statistics for the Wizard and CMU DoG datasets.
- Table 7 reports the statistics of the Wizard and CMU DoG datasets.
B Comparison with DialoGPT
The comparison tests whether a strong pre-trained dialogue model remains sufficient after fine-tuning on knowledge-grounded dialogues without associated knowledge. Results indicate that external knowledge remains necessary, although the advantage narrows on CMU DoG because many conversations weakly correlate with their documents.
- The comparison fine-tunes DialoGPT on knowledge-grounded dialogues after discarding the associated knowledge.
- DialoGPT uses the OpenAI GPT-2 model with 345M parameters because it performed best in the original evaluation.
- External knowledge remains necessary even when a powerful pre-trained language model is used for dialogue generation.
- In CMU DoG, the gap between DialoGPT and KnowledGPT narrows because about 35% of conversations weakly correlate with the document.
C Impact of Maximum Tokens of GPT-2
The experiment examines how GPT-2trunc behaves as its maximum token limit changes while ground-truth knowledge is included. Retaining ground-truth knowledge helps, but additional noise harms performance.
- The experiment gradually increases GPT-2trunc’s maximum token constraint while retaining ground-truth knowledge in the input.
- Increasing the maximum token limit introduces more irrelevant knowledge into the input.
- Ground-truth knowledge makes GPT-2trunc consistently better than randomly mixing candidates, which risks cutting the ground-truth.
- Even when ground-truth knowledge is retained, introducing more noise affects performance negatively.
D Impact of the Size of GPT-2
Replacing GPT-2 (117M) with GPT-2 (345M) reduces perplexity but does not significantly improve F1, so the smaller model is favored for balancing efficacy and cost.
- The study evaluates whether KnowledGPT benefits from replacing GPT-2 with a larger model.
- GPT-2 (345M) further reduces PPL but does not significantly improve F1 over GPT-2 (117M).The authors attribute this possibly to less accurate feedback from the larger model to the knowledge selection module during learning.
- GPT-2 (117M) remains favored in practice to balance efficacy and cost.
E Case Study
Case studies from Wizard of Wikipedia show KnowledGPT selecting knowledge more accurately and using associated knowledge more effectively with its selection, reinforcement-learning, and curriculum-learning components.
- Case Study: The examples are drawn from Test Seen and Test Unseen of Wizard of Wikipedia, pairing dialogue context with Wikipedia knowledge retrieved from the dialogue.
- Case Study: KnowledGPT more accurately locates relevant knowledge through its knowledge selection module and reinforcement learning.
- Case Study: Curriculum learning helps KnowledGPT make better use of associated knowledge.