TOD-BERT: Pre-trained Natural Language Understanding for Task-Oriented Dialogue

• Fine-tuning pre-trained models on conversational corpora can underperform because conversation and writing have intrinsically different linguistic patterns and data distributions.Source: §1 · p. 1
• Task-oriented dialogues contain explicit goals, user and system behaviors, and many interactions, but datasets are small and scattered because collection and labeling are time-consuming.Source: §1 · p. 1
• TOD-BERT combines nine human-human and multi-turn task-oriented dialogue corpora for task-oriented language-model pre-training.The unified corpora contain around 100k dialogues and 1.4M utterances across over 60 domains.Source: §1 · p. 1
• TOD-BERT adds user and system tokens and response contrastive learning to BERT-style masked-language-model pre-training.Source: §1 · p. 2
• TOD-BERT outperforms BERT, GPT-2, and DialoGPT on intention recognition, dialogue state tracking, dialogue act prediction, and response selection, with stronger few-shot ability.Source: §1 · p. 2

• GPT and GPT-2 use uni-directional Transformer decoders trained with left-to-right generation likelihood for language generation tasks.Source: §2 · p. 2
• UniLM supports both uni-directional and bi-directional attention, while CTRL conditions on control codes and T5 unifies multiple text-to-text modeling tasks.Source: §2 · p. 2
• Dialogue pre-trained language models have mostly used Reddit or Twitter data for open-domain response generation and retrieval.Source: §2 · p. 2
• Task-oriented pre-training has few related works, including GPT-2 trained on one dataset for response generation and Reddit-pre-trained response selection models.Source: §2 · p. 2
• Table 1 presents statistics for the task-oriented dialogue datasets used in the paper.Source: Table 1 · p. 3

3.1 Datasets

• The authors collect nine English, human-human, multi-turn task-oriented datasets containing 100,707 dialogues and 1,388,152 utterances across over 60 domains.Source: §3.1 · p. 3
• The dataset collection includes varied resources such as MetaLWOZ, Schema, Taskmaster, MWOZ, MSR-E2E, SMD, Frames, WOZ, and CamRest676.Source: §3.1 · p. 3

3.2 TOD-BERT

• TOD-BERT uses BERT’s Transformer self-attention encoder with masked language modeling and response contrastive loss.Source: §3.2 · p. 4
• [USR] prefixes user utterances and [SYS] prefixes system responses before all dialogue utterances are concatenated into one sequence.Source: §3.2 · p. 4
• Dynamic masked language modeling predicts randomly masked tokens, while response contrastive learning uses dialogue contexts and responses with other batch responses as negatives.Source: §3.2 · p. 4
• The overall pre-training loss is the weighted sum of masked language-modeling and response-contrastive losses, summed directly in the experiments.Source: §3.2 · p. 5

• The evaluation covers intent recognition, dialogue state tracking, dialogue act prediction, and response selection.These are described as four crucial downstream tasks and core components in modularized task-oriented systems.Source: §4 · p. 5

Intent recognition

• Intent recognition predicts one class among I possible intents from an input sentence U.The model uses a pre-trained language model's [CLS] representation and trains a linear classifier with cross-entropy loss.Source: §4 · p. 5

Dialogue state tracking

• Dialogue state tracking predicts slot values for each domain-slot pair at every dialogue turn using the dialogue history.Each candidate slot value is represented by the pre-trained model, and cosine similarities produce distributions trained with summed cross-entropy.Source: §4 · p. 5

Dialogue act prediction

• Dialogue act prediction is multi-label classification because a system response may contain multiple acts simultaneously.The model predicts a binary result for every possible act, using sigmoid outputs and binary cross-entropy loss; an act is triggered when Ai > 0.5.Source: §4 · p. 5

Response selection

• Response selection ranks candidate system responses by computing similarity scores between a source context X and each response Yi.The dual encoder uses the most recent 256 context tokens and randomly sampled system responses as negative samples.Source: §4 · pp. 5–6

• The evaluation uses OOS, DSTC2, GSIM, and MWOZ, while excluding MWOZ test dialogues from pretraining.The first three evaluation corpora are not included in the pretrained task-oriented datasets.Source: §5 · p. 6

OOS

• OOS contains 15,100/3,100/5,500 train, validation, and test samples covering 151 intent classes across ten domains.The dataset includes 150 in-scope intents and one out-of-scope intent.Source: §5 · p. 6

DSTC2

• DSTC2 contains 1,612/506/1,117 train, validation, and test dialogues and uses 19 universal system dialogue acts.It is a human-machine task-oriented dataset that may include some system response noise.Source: §5 · p. 6

GSIM

• GSIM contains 1,500/469/1,039 train, validation, and test dialogues from combined movie and restaurant domains.It is a human-rewritten machine-machine corpus collected through dialogue self-play and crowdsourcing.Source: §5 · p. 6

MWOZ

• MWOZ contains 8,420/1,000/1,000 train, validation, and test dialogues across seven domains, with 30 domain-slot pairs tracked in testing.The evaluation uses MWOZ 2.1, which retains the transcripts but has cleaner state-label annotations.Source: §5 · p. 6

Probing

• TOD-BERT-jnt achieves the highest probing performance for domain classification, intent identification, and dialogue act prediction.The probing setup uses a single-layer perceptron on fixed pretrained representations.Source: §6.1 · p. 6

Intent Recognition

• 13.2% all-intent and 16.3% in-domain accuracy improvements over BERT occur for TOD-BERT-jnt in the 1-shot setting.The OOS evaluation also reports highest in-scope and out-of-scope accuracy for TOD-BERT-jnt.Source: §6.2 · pp. 6–7

Dialogue State Tracking

• 2.4% joint goal accuracy improvement over BERT is achieved by TOD-BERT-jnt on MWOZ 2.1.TOD-BERT outperforms BERT in the reported 1%, 5%, 10%, and 25% data settings.Source: §6.3 · p. 7

Dialogue Act Prediction

• 3.5% micro-F1 and 6.6% macro-F1 improvements over BERT are achieved by TOD-BERT-mlm on MWOZ with 1% of the training data.In full-data experiments, TOD-BERT consistently outperforms BERT and other baselines across datasets and metrics.Source: §6.4 · p. 8

Response Selection

• 65.8% 1-to-100 and 87.0% 3-to-100 accuracy are achieved by TOD-BERT-jnt on MWOZ, surpassing BERT by 18.3% and 11.5%.Similar advantages are observed on DSTC2 and GSIM, especially in few-shot settings.Source: §6.5 · p. 9, Table 6 · p. 9
• Response contrastive learning is beneficial, while response-selection results are sensitive to training batch size.The evaluation uses random batches of 100 examples and averages results across five random seeds.Source: §6.5 · p. 9

• TOD-BERT-jnt has clearer representation group boundaries than TOD-BERT-mlm, and both outperform BERT visually.The comparison uses system-response embeddings from the MWOZ test set, with colors denoting domain and dialogue-act labels.Source: §7 · p. 9
• K-means clustering evaluates whether BERT and TOD-BERT embeddings align with actual MWOZ domain labels using normalized mutual information.The analysis sets K to 10 and 20 before comparing clustering assignments with true domain labels.Source: §7 · p. 9

• TOD-BERT is trained on nine human-human and multi-turn task-oriented datasets spanning over 60 domains.The paper presents this broad training coverage as the basis for a task-oriented dialogue language model.Source: §8 · p. 9
• TOD-BERT outperforms BERT on intention classification, dialogue state tracking, dialogue act prediction, and response selection.These are the four dialogue downstream tasks identified in the conclusion.Source: §8 · p. 9
• TOD-BERT has a clear advantage in few-shot experiments when only limited labeled data is available.The conclusion also describes the model as easy to deploy and intended for application or fine-tuning on task-oriented conversational problems.Source: §8 · p. 9

• Figure 3 colors tSNE system-response representations by domain in the MWOZ test set.Source: Figure 3 · p. 12
• Figure 4 colors tSNE system-response representations by dialogue act in the MWOZ test set.Source: Figure 4 · p. 12
• Figure 5 colors tSNE system-response representations by dialogue slot in the MWOZ test set.Source: Figure 5 · p. 13