Source-linked AI summary
TOD-BERT: Pre-trained Natural Language Understanding for Task-Oriented Dialogue
Chien-Sheng Wu, Steven Hoi, Richard Socher, Caiming Xiong
TL;DR
Existing pre-trained language models are less useful for task-oriented dialogue because general text and task-oriented dialogue have different linguistic patterns. The paper unifies nine task-oriented dialogue datasets and pre-trains TOD-BERT with speaker tokens and response contrastive learning. TOD-BERT outperforms BERT and other strong baselines across four downstream applications and has stronger few-shot ability.
Problem
Different linguistic patterns between general text and task-oriented dialogue make existing pre-trained language models less useful for task-oriented dialogue.
Method
TOD-BERT unifies nine human-human and multi-turn task-oriented dialogue datasets and adds user/system tokens plus response contrastive learning to masked language-model pre-training.
Results
TOD-BERT outperforms BERT and other strong baselines on intention recognition, dialogue state tracking, dialogue act prediction, and response selection.
Takeaways & Limitations
TOD-BERT has stronger few-shot ability, suggesting it can reduce the need for expensive human-annotated labels in task-oriented dialogue.
Takeaways & Limitations
Response selection performance is sensitive to batch size, which may be limited by hardware.
Abstract
from arXiv · showhide
The underlying difference of linguistic patterns between general text and task-oriented dialogue makes existing pre-trained language models less useful in practice. In this work, we unify nine human-human and multi-turn task-oriented dialogue datasets for language modeling. To better model dialogue behavior during pre-training, we incorporate user and system tokens into the masked language modeling. We propose a contrastive objective function to simulate the response selection task. Our pre-trained task-oriented dialogue BERT (TOD-BERT) outperforms strong baselines like BERT on four downstream task-oriented dialogue applications, including intention recognition, dialogue state tracking, dialogue act prediction, and response selection. We also show that TOD-BERT has a stronger few-shot ability that can mitigate the data scarcity problem for task-oriented dialogue.
1 Introduction
Task-oriented dialogue differs from general text and is scarce, motivating TOD-BERT, which pre-trains on unified dialogue corpora with speaker-aware and response-selection objectives. It outperforms strong baselines across four downstream tasks and shows stronger few-shot ability.
- Fine-tuning pre-trained models on conversational corpora can underperform because conversation and writing have intrinsically different linguistic patterns and data distributions.
- Task-oriented dialogues contain explicit goals, user and system behaviors, and many interactions, but datasets are small and scattered because collection and labeling are time-consuming.
- TOD-BERT combines nine human-human and multi-turn task-oriented dialogue corpora for task-oriented language-model pre-training.The unified corpora contain around 100k dialogues and 1.4M utterances across over 60 domains.
- TOD-BERT adds user and system tokens and response contrastive learning to BERT-style masked-language-model pre-training.
- TOD-BERT outperforms BERT, GPT-2, and DialoGPT on intention recognition, dialogue state tracking, dialogue act prediction, and response selection, with stronger few-shot ability.
2 Related Work
Prior pre-trained language models primarily target general text or open-domain dialogue, while task-oriented pre-training has comparatively few related efforts and limited dataset coverage.
- GPT and GPT-2 use uni-directional Transformer decoders trained with left-to-right generation likelihood for language generation tasks.
- UniLM supports both uni-directional and bi-directional attention, while CTRL conditions on control codes and T5 unifies multiple text-to-text modeling tasks.
- Dialogue pre-trained language models have mostly used Reddit or Twitter data for open-domain response generation and retrieval.
- Task-oriented pre-training has few related works, including GPT-2 trained on one dataset for response generation and Reddit-pre-trained response selection models.
- Table 1 presents statistics for the task-oriented dialogue datasets used in the paper.
3 Method
The method unifies task-oriented dialogue datasets and adapts BERT with speaker tokens, masked language modeling, and response contrastive learning. The resulting objectives encode dialogue behavior and response similarity without additional annotation.
- 3.1 Datasets: The authors collect nine English, human-human, multi-turn task-oriented datasets containing 100,707 dialogues and 1,388,152 utterances across over 60 domains.
- 3.1 Datasets: The dataset collection includes varied resources such as MetaLWOZ, Schema, Taskmaster, MWOZ, MSR-E2E, SMD, Frames, WOZ, and CamRest676.
- 3.2 TOD-BERT: TOD-BERT uses BERT’s Transformer self-attention encoder with masked language modeling and response contrastive loss.
- 3.2 TOD-BERT: [USR] prefixes user utterances and [SYS] prefixes system responses before all dialogue utterances are concatenated into one sequence.
- 3.2 TOD-BERT: Dynamic masked language modeling predicts randomly masked tokens, while response contrastive learning uses dialogue contexts and responses with other batch responses as negatives.
- 3.2 TOD-BERT: The overall pre-training loss is the weighted sum of masked language-modeling and response-contrastive losses, summed directly in the experiments.
4 Downstream Tasks
The paper evaluates TOD-BERT across four core task-oriented dialogue understanding tasks, each formulated with a task-specific prediction objective. These include intent recognition, dialogue state tracking, dialogue act prediction, and response selection.
- The evaluation covers intent recognition, dialogue state tracking, dialogue act prediction, and response selection.These are described as four crucial downstream tasks and core components in modularized task-oriented systems.
- Intent recognition: Intent recognition predicts one class among I possible intents from an input sentence U.The model uses a pre-trained language model's [CLS] representation and trains a linear classifier with cross-entropy loss.
- Dialogue state tracking: Dialogue state tracking predicts slot values for each domain-slot pair at every dialogue turn using the dialogue history.Each candidate slot value is represented by the pre-trained model, and cosine similarities produce distributions trained with summed cross-entropy.
- Dialogue act prediction: Dialogue act prediction is multi-label classification because a system response may contain multiple acts simultaneously.The model predicts a binary result for every possible act, using sigmoid outputs and binary cross-entropy loss; an act is triggered when Ai > 0.5.
- Response selection: Response selection ranks candidate system responses by computing similarity scores between a source context X and each response Yi.The dual encoder uses the most recent 256 context tokens and randomly sampled system responses as negative samples.
5 Evaluation Datasets
The downstream evaluation uses OOS, DSTC2, GSIM, and MWOZ, with safeguards against pretraining-test contamination for MWOZ. These datasets span intent, dialogue-act, and state-tracking settings.
- The evaluation uses OOS, DSTC2, GSIM, and MWOZ, while excluding MWOZ test dialogues from pretraining.The first three evaluation corpora are not included in the pretrained task-oriented datasets.
- OOS: OOS contains 15,100/3,100/5,500 train, validation, and test samples covering 151 intent classes across ten domains.The dataset includes 150 in-scope intents and one out-of-scope intent.
- DSTC2: DSTC2 contains 1,612/506/1,117 train, validation, and test dialogues and uses 19 universal system dialogue acts.It is a human-machine task-oriented dataset that may include some system response noise.
- GSIM: GSIM contains 1,500/469/1,039 train, validation, and test dialogues from combined movie and restaurant domains.It is a human-rewritten machine-machine corpus collected through dialogue self-play and crowdsourcing.
- MWOZ: MWOZ contains 8,420/1,000/1,000 train, validation, and test dialogues across seven domains, with 30 domain-slot pairs tracked in testing.The evaluation uses MWOZ 2.1, which retains the transcripts but has cleaner state-label annotations.
6 Results
TOD-BERT is evaluated through probing and four task-oriented dialogue applications, including few-shot settings. It consistently outperforms BERT and other baselines, with response contrastive learning and limited-data evaluations highlighting its strengths.
- Probing: TOD-BERT-jnt achieves the highest probing performance for domain classification, intent identification, and dialogue act prediction.The probing setup uses a single-layer perceptron on fixed pretrained representations.
- Intent Recognition: 13.2% all-intent and 16.3% in-domain accuracy improvements over BERT occur for TOD-BERT-jnt in the 1-shot setting.The OOS evaluation also reports highest in-scope and out-of-scope accuracy for TOD-BERT-jnt.
- Dialogue State Tracking: 2.4% joint goal accuracy improvement over BERT is achieved by TOD-BERT-jnt on MWOZ 2.1.TOD-BERT outperforms BERT in the reported 1%, 5%, 10%, and 25% data settings.
- Dialogue Act Prediction: 3.5% micro-F1 and 6.6% macro-F1 improvements over BERT are achieved by TOD-BERT-mlm on MWOZ with 1% of the training data.In full-data experiments, TOD-BERT consistently outperforms BERT and other baselines across datasets and metrics.
- Response Selection: 65.8% 1-to-100 and 87.0% 3-to-100 accuracy are achieved by TOD-BERT-jnt on MWOZ, surpassing BERT by 18.3% and 11.5%.Similar advantages are observed on DSTC2 and GSIM, especially in few-shot settings.
- Response Selection: Response contrastive learning is beneficial, while response-selection results are sensitive to training batch size.The evaluation uses random batches of 100 examples and averages results across five random seeds.
7 Visualization
TOD-BERT produces more clearly separated representations than BERT, with joint masked-language-modeling and contrastive pretraining yielding the clearest group boundaries and higher clustering alignment.
- TOD-BERT-jnt has clearer representation group boundaries than TOD-BERT-mlm, and both outperform BERT visually.The comparison uses system-response embeddings from the MWOZ test set, with colors denoting domain and dialogue-act labels.
- K-means clustering evaluates whether BERT and TOD-BERT embeddings align with actual MWOZ domain labels using normalized mutual information.The analysis sets K to 10 and 20 before comparing clustering assignments with true domain labels.
8 Conclusion
TOD-BERT is a task-oriented dialogue BERT trained across diverse dialogue corpora and designed for broad downstream use. It outperforms BERT across four dialogue tasks and remains advantageous in few-shot settings.
- TOD-BERT is trained on nine human-human and multi-turn task-oriented datasets spanning over 60 domains.The paper presents this broad training coverage as the basis for a task-oriented dialogue language model.
- TOD-BERT outperforms BERT on intention classification, dialogue state tracking, dialogue act prediction, and response selection.These are the four dialogue downstream tasks identified in the conclusion.
- TOD-BERT has a clear advantage in few-shot experiments when only limited labeled data is available.The conclusion also describes the model as easy to deploy and intended for application or fine-tuning on task-oriented conversational problems.
A Appendices
The appendix visualizes TOD-BERT and BERT system-response representations under three labeling schemes: domains, dialogue acts, and dialogue slots.
- Figure 3 colors tSNE system-response representations by domain in the MWOZ test set.
- Figure 4 colors tSNE system-response representations by dialogue act in the MWOZ test set.
- Figure 5 colors tSNE system-response representations by dialogue slot in the MWOZ test set.