Source-linked AI summary
PLATO-2: Towards Building an Open-Domain Chatbot via Curriculum Learning
Siqi Bao, Huang He, Fan Wang, Hua Wu, Haifeng Wang, Wenquan Wu, Zhen Guo, Zhibin Liu, Xinchao Xu
TL;DR
Open-domain dialogue requires modeling multiple appropriate responses while maintaining response quality and coherence. PLATO-2 addresses this with curriculum learning: a coarse one-to-one generator is followed by latent-variable generation and evaluation models. It achieves substantial improvements and state-of-the-art results across Chinese and English evaluations.
Problem
Open-domain dialogue involves a one-to-many mapping in which one context can have multiple appropriate responses, while directly scaling such modeling causes training instability and efficiency issues.
Method
PLATO-2 trains a coarse one-to-one generation model first, then trains a latent-variable fine-grained generator and an evaluation model for diverse responses and coherence estimation.
Results
PLATO-2 achieves substantial improvements over state-of-the-art methods in both Chinese and English evaluations, including new state-of-the-art results.
Takeaways & Limitations
Curriculum learning supports scaling PLATO-2 to billions of parameters and adapting the two-stage models to multiple conversational tasks.
Abstract
from arXiv · showhide
To build a high-quality open-domain chatbot, we introduce the effective training process of PLATO-2 via curriculum learning. There are two stages involved in the learning process. In the first stage, a coarse-grained generation model is trained to learn response generation under the simplified framework of one-to-one mapping. In the second stage, a fine-grained generative model augmented with latent variables and an evaluation model are further trained to generate diverse responses and to select the best response, respectively. PLATO-2 was trained on both Chinese and English data, whose effectiveness and superiority are verified through comprehensive evaluations, achieving new state-of-the-art results.
1 Introduction
PLATO-2 scales PLATO through curriculum learning, progressing from one-to-one response generation to latent-variable modeling and response evaluation. Trained in Chinese and English, it achieves state-of-the-art performance across comprehensive evaluations.
- 1 Introduction: PLATO-2 addresses the instability and efficiency difficulties encountered when directly scaling PLATO's one-to-many dialogue modeling.PLATO models one-to-many relationships with discrete latent variables, but learning this relationship from scratch complicates direct scaling.
- 1 Introduction: PLATO-2 uses curriculum learning to progress from coarse-grained one-to-one response generation to fine-grained one-to-many modeling.The second stage adds a latent-variable generation model for diverse responses and an evaluation model for response coherence.
- 1 Introduction: PLATO-2 achieves new state-of-the-art results and adapts to task-oriented and knowledge-grounded dialogue.The two-stage framework enabled scaling to billions of parameters and benefited multiple conversational tasks.
- 1 Introduction: Comprehensive experiments show that PLATO-2 outperforms Meena, Blender, and other state-of-the-art models on Chinese and English evaluations.The released models include English and Chinese variants trained with different model sizes and large-scale conversation data.
2 Methodology
PLATO-2 uses curriculum learning to progress from coarse one-to-one response generation to fine-grained one-to-many generation and response evaluation. Its unified transformer architecture supports diverse candidate generation and coherence-based response selection.
- 2 Methodology: PLATO-2 trains a coarse-grained baseline first, then fine-grained generation and evaluation models in a two-stage curriculum.Stage 1 learns general response generation under one-to-one mapping; stage 2 models one-to-many generation and response coherence.
- 2 Methodology: The fine-grained generator introduces a K-way discrete latent variable representing response speech acts and jointly learns latent-act recognition and response generation.The model estimates p(z|c,r) and generates responses with p(r|c,z); Gumbel-Softmax approximates nondifferentiable sampling.
- 2 Methodology: The fine-grained generation objective combines autoregressive response modeling with a bag-of-words loss that encourages latent variables to capture global response information.The bag-of-words loss discards word order while predicting target-response words non-autoregressively.
- 2 Methodology: The evaluation model estimates bidirectional context-response coherence and selects the candidate with the highest coherence value.Candidates are generated under each latent value, then ranked by p(l_r=1|c,r).
- 2 Methodology: The evaluation network additionally trains a masked language model, randomly masking 15% of input tokens and recovering them to maintain distributed representations.The inputs include dialogue context and responses, with masked tokens predicted from the remaining tokens.
3 Experiments
The experiments compare PLATO-2 with established dialogue systems using English and Chinese data, across automatic, human, interactive, and static evaluations. Results show strong performance for PLATO-2, including advantages over Blender and XiaoIce in the reported evaluations.
- Evaluation Metrics: Automatic evaluation measures lexical diversity with distinct-1/2, while human evaluation scores coherence, informativeness, engagingness, and humanness on a 0–2 scale.The experiments include both static and interactive evaluations, with self-chat logs assessed by crowd workers.
- Experimental Setup: The experiments compare models across three groups matched by parameter scale and training data, including PLATO, DialoGPT, Blender, and multiple PLATO-2 sizes.The groups compare similar-scale systems, while the largest group contrasts Blender 2.7B with PLATO-2 1.6B.
- Self-Chat Evaluation: PLATO-2 achieves the best performance across human and automatic self-chat evaluations, outperforming Blender in the largest comparison group.The comparison also reports stronger lexical diversity than Blender according to distinct-1/2.
- Chinese Interactive Evaluation: PLATO-2 achieves significant improvements over XiaoIce across all human evaluation metrics in Chinese interactive evaluation.XiaoIce obtains higher distinct values, which may reflect retrieval-based response generation.
3.5 Discussions
The discussion contrasts Blender’s rapid topic switching with PLATO-2’s sustained, in-depth discussions, and attributes the difference to diverse generation plus coherent response selection. Further evaluations associate the two-stage curriculum with strong performance across conversational tasks.
- 3.5.1 Case Analysis: Blender tends to switch topics quickly, whereas PLATO-2 can maintain the original topic for deeper discussion.The differing discourse styles are illustrated through self-chat examples; Blender’s style may relate to BST fine-tuning data.
- 3.5.1 Case Analysis: PLATO-2 sustains the start topic and conducts more in-depth discussions than Blender.The comparison is reported in the context of Table 4’s start-topic evaluation.
- 3.5.1 Case Analysis: PLATO-2’s diverse responses and evaluation model are identified as reasons it generates coherent, topic-focused discussions.The discrete latent variable models one-to-many responses, while the evaluation model selects a coherent response.
- 3.5.2 Ablation Study: Table 5 evaluates the separate effects of PLATO-2’s two-stage models and supports the effectiveness of curriculum learning.The analysis separates the fine-grained generation and evaluation components within the curriculum framework.
- 3.5.3 Further Exploration of PLATO-2: PLATO-2 achieved first place in DSTC9 tasks covering open-domain, knowledge-grounded, and task-oriented conversation.The authors present this as evidence that the two-stage framework adapts across conversational tasks.
4 Related Work
Related work covers large-scale pretrained language models and their application to open-domain dialogue generation. GPT-2 scaled transformer language modeling substantially, while Megatron-LM demonstrated even larger model training through model parallelism.
- Large-scale Language Models: The related-work discussion situates open-domain dialogue generation within broader progress in large-scale language models.GPT and BERT are presented as representative unidirectional and bidirectional pretrained models.
- Large-scale Language Models: GPT-2 extended pretrained language-model scale from 117M to 1.5B parameters.The work modified pre-normalization and weight initialization to support this scale increase.
- Large-scale Language Models: Megatron-LM trained an 8.3B-parameter model on 512 GPUs to address memory constraints.It used model parallelism for large-scale training.
5 Conclusion
PLATO-2 uses curriculum learning to train coarse-grained generation first, then fine-grained generation and evaluation models. Experiments report substantial improvements over state-of-the-art methods in both Chinese and English evaluations.
- 5 Conclusion: PLATO-2’s two-stage curriculum trains general response generation before diverse generation and response-coherence estimation.The first stage uses one-to-one mapping, while the second trains fine-grained generation and evaluation models.
- 5 Conclusion: PLATO-2 achieved substantial improvements over state-of-the-art methods in both Chinese and English evaluations.The conclusion reports this outcome across the paper’s evaluations.
A Data Cleaning Process
PLATO-2 constructs training examples from social-media message trees and removes messages or subtrees that fail length, formatting, duplication, content, or source-quality checks.
- A Data Cleaning Process: Each conversation path from a message-tree root to a node becomes a training sample, with prior turns as context and the node as response.The data consist of English and Chinese open-domain social-media conversations.
- A Data Cleaning Process: Messages are removed when token length is outside the 2–128 BPE-token range.This is one of the explicit cleaning conditions.
- A Data Cleaning Process: Cleaning removes messages with long words or texts, low alphabetic content, URLs, special strings, parent-text overlap, or excessive repetition.These conditions target malformed, redundant, or unsuitable messages.
- A Data Cleaning Process: Cleaning also removes offensive messages, quarantined subreddits, and messages authored by known bots.These filters constrain source and content quality in the English training data.
B Training Configurations
PLATO-2 is provided in three model sizes, with configurations differing in parameter count, transformer depth, attention heads, and embedding dimension.
- PLATO-2 has standard, small, and tiny versions with 1.6B, 314M, and 93M parameters, respectively.
- The standard 1.6B model uses 32 transformer blocks, 32 attention heads, and a 2048-dimensional embedding.
- The small and tiny models use 24 and 12 transformer blocks, 16 and 12 attention heads, and embedding dimensions of 1024 and 768, respectively.
C Chinese Case Analysis
The Chinese case analysis compares human-bot interactions from Microsoft XiaoIce and PLATO-2 using chat examples shown in Figure 4.
- Figure 4 presents human-bot chat examples from Microsoft XiaoIce and PLATO-2, with original logs on the left and translated logs on the right.
- XiaoIce produces some responses that are incoherent with the context and includes abrupt topic changes.
- The PLATO-2 interaction is described as more coherent and engaging than the XiaoIce examples.
D Scoring Criteria in Human Evaluation
The paper reports the criteria and score details used for human evaluation, with the criteria provided in Table 7.
- The human evaluation criteria are provided in Table 7.
- The human-evaluation criteria are documented alongside the score details in the reported evaluation materials.
- Table 7 contains score details for four metrics used in human evaluation.
E Response Selection Comparison
The response-selection experiments construct an annotated Chinese dataset and compare distinct scoring functions using ranking metrics.
- The annotated Chinese response-selection dataset contains 100 test-set contexts, each paired with 10 candidate responses retrieved using a commercial chatbot.
- Three 336M-parameter scoring models are trained, including forward response-generation probability, reverse probability p(c|r), and bidirectional coherence probability p(lr|c, r).
- PLATO-2’s evaluation model is reported as better at selecting appropriate responses than the compared scoring functions.
- Table 8 summarizes results using mean average precision, mean reciprocal rank, and precision at position 1.