Source-linked AI summary
A Large-Scale Chinese Short-Text Conversation Dataset
Yida Wang, Pei Ke, Yinhe Zheng, Kaili Huang, Yong Jiang, Xiaoyan Zhu, Minlie Huang
TL;DR
Large-scale high-quality Chinese dialogue corpora are scarce, limiting the development of Chinese dialogue generation models. The paper introduces cleaned LCCC datasets and associated pre-training models, using rule- and classifier-based filtering; the resources are released for research on short-text conversation modeling.
Problem
Large-scale high-quality Chinese dialogue corpora are scarce, hindering the development of Chinese dialogue generation models.
Method
The paper constructs LCCC using rule-based and classifier-based filtering and releases pre-training Chinese dialogue models trained on the resulting data.
Results
LCCC contains 6.8M conversations in its base version and 12M in its large version, with models and datasets released for public use.
Takeaways & Limitations
LCCC and its released models provide resources for research on short-text and open-domain Chinese conversation generation.
Abstract
from arXiv · showhide
The advancements of neural dialogue generation models show promising results on modeling short-text conversations. However, training such models usually needs a large-scale high-quality dialogue corpus, which is hard to access. In this paper, we present a large-scale cleaned Chinese conversation dataset, LCCC, which contains a base version (6.8million dialogues) and a large version (12.0 million dialogues). The quality of our dataset is ensured by a rigorous data cleaning pipeline, which is built based on a set of rules and a classifier that is trained on manually annotated 110K dialogue pairs. We also release pre-training dialogue models which are trained on LCCC-base and LCCC-large respectively. The cleaned dataset and the pre-training models will facilitate the research of short-text conversation modeling. All the models and datasets are available at https://github.com/thu-coai/CDial-GPT.
1 Introduction
LCCC addresses the shortage of large-scale Chinese dialogue corpora by constructing a cleaned dataset and releasing pre-training models for Chinese dialogue generation.
- 1 Introduction: LCCC provides a large-scale cleaned Chinese conversation dataset in base and large versions, addressing the shortage of Chinese dialogue corpora.The dataset is filtered from Weibo and other Chinese corpora.
- 1 Introduction: A two-phase pipeline combines heuristic rules with classifiers trained on manually labeled data to improve dialogue quality.The classifiers perform additional filtering after rule-based removal of inappropriate content.
- 1 Introduction: The authors release pre-training models for Chinese dialogue generation alongside the datasets.The models are first pre-trained on a Chinese novel dataset and then post-trained on LCCC.
- 1 Introduction: LCCC is intended as a benchmark for open-domain Chinese conversation generation, with models and data released for public use.The paper also conducts experiments to evaluate Chinese dialogue generation performance.
2 Related work
Prior dialogue datasets draw on large public resources or smaller crowd-sourced collections, while Chinese dialogue generation has also begun using pre-trained language models.
- 2 Related work: Public resources such as movie scripts, social media, and technical forums provide large-scale dialogue corpora for open-domain systems.Examples include OpenSubtitles, Twitter, Reddit, and Weibo.
- 2 Related work: Crowd-sourced datasets are generally cleaner but smaller and target specialized tasks such as knowledge grounding, persona modeling, or emotional dialogue.Examples include Wizard of Wikipedia, PERSONA-CHAT, and DailyDialog.
- 2 Related work: Prior pre-training work applies large English conversation corpora or smaller Chinese dialogue datasets to dialogue generation.DialoGPT uses Reddit conversations, while other work fine-tunes Chinese GPT models on smaller dialogue datasets.
3 Datasets
The dataset pipeline constructs LCCC from Weibo and additional Chinese corpora, applying rule-based and classifier-based filtering to produce cleaned conversation resources and compare them with existing datasets.
- 3 Datasets: 79M Weibo conversations are cleaned into LCCC-base, then combined with public Chinese conversation datasets to form LCCC-large under more relaxed classifier thresholds.LCCC-large incorporates both single-turn and multi-turn corpora from multiple open-source repositories.
- 3 Datasets: The collection process selects seed users associated with professional news accounts and reconstructs root-to-leaf comment paths as conversation sessions.A Depth First Search process yields 79 million raw conversation sessions.
- Rule-based Noise Filtering: Rule-based filtering removes platform artifacts, URLs, excessive turns or repetition, unsuitable response lengths, advertisements, generic responses, and post-response duplicates.The rules target both structural noise and low-quality response patterns.
- Rule-based Noise Filtering: Blacklist filtering removes conversations containing dirty or sensitive words, special-topic terms, unknown abbreviations, symbols, emoji, and platform-related signs.These blacklists supplement the other rule-based noise filters.
- 3 Datasets: Classifier-based filtering uses BERT models trained on 100,000 labeled conversations and 10,000 labeled utterances to detect semantic, grammatical, contextual, and relevance problems.The reported test accuracies are 73.76% and 77.60% for the two classifiers.
- 3.3 Statistics and Results: Dataset statistics and comparisons are reported for LCCC-base, LCCC-large, and existing Chinese conversation datasets.Table 2 presents statistics for the two LCCC versions, while Table 3 compares LCCC with existing datasets.
4 Models
The paper uses GPT-style transformer decoders for Chinese dialogue generation, representing conversation history with word, speaker, and position embeddings. Models are post-trained on LCCC-base or LCCC-large, with variants based on GPT and GPT2 architectures.
- Architecture: GPT-style transformer decoders generate responses autoregressively using masked multi-head self-attention over left-context information.At each time step, the decoder observes only preceding tokens.
- Input Representation: Conversation history is concatenated into one sequence whose input combines word, speaker, and position embeddings.Speaker symbols separate participants, while [CLS] and [SEP] mark sequence boundaries.
- Post-training: The models are post-trained from a pre-trained Chinese GPT model on multi-turn LCCC conversations, treating each later utterance as a response to preceding history.This produces training instances from every sentence after the first in a dialogue.
- Model Variants: The released variants include GPT and GPT2 models trained with LCCC-base or LCCC-large, using AdamW, Noam learning-rate decay, and 12 attention heads.All models use 12 layers and 768-dimensional word embeddings.
5 Experiments
The experiments fine-tune and compare Chinese dialogue models using automatic metrics, human judgments, and generated examples. Models trained on LCCC achieve the strongest overall evaluation results, despite lower BLEU-2 than Transformer.
- 5.1 Fine-tuning Settings and Implementation Details: The study compares LCCC-based models with GPT2-chitchat, vanilla Transformer, and attention-based sequence-to-sequence baselines.Transformer and Attn-Seq2Seq are trained on STC without pre-training, while GPT2-chitchat is pre-trained on 500,000 Chinese conversations.
- Case Study: Generated examples include interactive and self-playing multi-turn dialogues, alongside cases from CDialGPTLCCC−large fine-tuned on STC.The examples are decoded with top-p sampling at temperature 0.7.
- 5.2 Evaluation: The evaluation uses BLEU, distinct n-grams, Greedy Matching, Embedding Average, and perplexity to assess generation quality from multiple perspectives.Human judges additionally score fluency, relevance, and informativeness using a 2/1/0 scheme.
- 5.2 Evaluation: LCCC-trained models achieve the best perplexity and outperform other models in Greedy Matching and Embedding Average, although Transformer has higher BLEU-2.The paper attributes GPT2-chitchat’s highest distinct scores alongside poor generation quality to its smaller model scale.
- Human Evaluation: Human evaluation shows that models trained on LCCC outperform the alternatives, with LCCC-base slightly exceeding Transformer despite weaker automatic metrics.Fleiss kappa ranges from 0.39 to 0.44, indicating fair-to-moderate annotation agreement.
6 Conclusion
The paper presents LCCC, a large-scale cleaned Chinese conversation dataset with two versions, and releases Chinese dialogue pre-training models. Its rule-based and classifier-based filtering procedures target higher-quality dialogue data, and the resources are publicly released.
- 6 Conclusion: LCCC provides a 6.8M-conversation base version and a 12M-conversation large version produced through filtering procedures.The filtering pipeline combines rules with classifiers, and the pre-training models use 12M open-domain conversations.
- 6 Conclusion: The paper releases pre-training models for Chinese dialogue generation together with the cleaned dataset for public use.The released resources are intended for research on Chinese dialogue generation.
- 6 Conclusion: The work uses rule-based and classifier-based filtering to obtain high-quality Chinese conversation data at large scale.This construction addresses the paper’s focus on cleaning large-scale dialogue data.