Source-linked AI summary
Multi-Task Pre-Training for Plug-and-Play Task-Oriented Dialogue System
Yixuan Su, Lei Shu, Elman Mansimov, Arshit Gupta, Deng Cai, Yi-An Lai, Yi Zhang
TL;DR
Existing TOD methods often use cascaded generation, which can accumulate errors and require extensive annotation. PPTOD unifies TOD subtasks with task-specific prompts and multi-task pre-training over heterogeneous, partially annotated dialogue corpora; it achieves state-of-the-art results across three benchmark tasks in high- and low-resource settings.
Problem
Existing TOD methods often formulate the task as cascaded generation, causing error accumulation across subtasks and requiring annotation for all subtasks.
Method
PPTOD is a unified plug-and-play model that uses task-specific prompts and multi-task pre-training on heterogeneous dialogue corpora with partial annotations.
Results
PPTOD achieves state-of-the-art results across three benchmark TOD tasks in both full-training and low-resource settings, according to automatic and human evaluations.
Takeaways & Limitations
PPTOD supports task-oriented dialogue understanding and response generation in a plug-and-play manner across high-resource and low-resource settings.
Takeaways & Limitations
The authors report that the large model is less capable of generating delexicalized tokens unseen during pre-training for NLG.
Abstract
from arXiv · showhide
Pre-trained language models have been recently shown to benefit task-oriented dialogue (TOD) systems. Despite their success, existing methods often formulate this task as a cascaded generation problem which can lead to error accumulation across different sub-tasks and greater data annotation overhead. In this study, we present PPTOD, a unified plug-and-play model for task-oriented dialogue. In addition, we introduce a new dialogue multi-task pre-training strategy that allows the model to learn the primary TOD task completion skills from heterogeneous dialog corpora. We extensively test our model on three benchmark TOD tasks, including end-to-end dialogue modelling, dialogue state tracking, and intent classification. Experimental results show that PPTOD achieves new state of the art on all evaluated tasks in both high-resource and low-resource scenarios. Furthermore, comparisons against previous SOTA methods show that the responses generated by PPTOD are more factually correct and semantically coherent as judged by human annotators.
1 Introduction
Existing TOD systems commonly use cascaded generation, tying later subtasks to earlier outputs and requiring broadly annotated training data. PPTOD instead unifies TOD modules and decouples their generation through task-specific prompts, while using multi-task pre-training on heterogeneous, partially annotated dialogue corpora.
- Cascaded TOD systems generate later subtasks from earlier outputs, including responses conditioned on dialogue state tracking and policy outputs.
- This formulation propagates earlier errors, requires annotation for all subtasks, excludes partially annotated data, and increases inference latency.
- PPTOD integrates dialogue state tracking, policy learning, and generation into one model, steering each subtask with a task-specific natural-language prompt.
- Decoupled subtask generation lets PPTOD learn from partially annotated data and generate different subtask outputs in parallel, reducing error accumulation and inference latency.
- The proposed pre-training strategy combines heterogeneous dialogue corpora with partial annotations for NLU, DST, POL, and NLG.
- PPTOD is evaluated on end-to-end dialogue modelling, dialogue state tracking, and intent classification in full-training and low-resource settings, with automatic and human evaluations.
2 Related Work
TOD research has progressed from pipelined systems toward neural and pre-trained models, but many pre-training approaches still depend on fully annotated data. This work uses one multi-task model across relevant TOD subtasks and leverages partially annotated supplementary corpora.
- Task-Oriented Dialogue: Traditional TOD systems pipeline belief-state tracking, policy learning, and response generation to accomplish user goals.
- Task-Oriented Dialogue: Neural systems increasingly address multiple TOD subtasks, while pre-trained models extend this approach to dialogue generation and broader TOD modelling.
- Task-Oriented Dialogue: Prior dialogue-pre-training methods require data fully annotated for DST, POL, and NLG, limiting the corpora they can use.
- Language Model Pre-training: Language-model pre-training has advanced language understanding and text generation across a wide range of NLP tasks.
- Language Model Pre-training: Dialogue pre-training has also used open-domain conversational data, while TOD-specific pre-training has targeted dialogue understanding tasks.
- Pre-training on Supplementary Data: PPTOD differs from supplementary-training work by using a single multi-task model for all relevant TOD subtasks.
3 Methodology
PPTOD uses a unified text-generation formulation for multiple TOD tasks, trained through dialogue multi-task pre-training on heterogeneous, partially annotated corpora and adapted with the same objective.
- Dialogue Multi-Task Pre-training: The pre-training corpus combines eleven human-written multi-turn dialogue corpora containing partially annotated data for TOD-related tasks.The collected corpora contain over 2.3M utterances across 80 domains.
- Unified Task Formulation: PPTOD casts NLU, DST, POL, and NLG as a common plug-and-play text-generation problem selected by task-specific prompts.Prompts map tasks to user intent, belief state, dialogue act, or system response outputs.
- Training Samples: Each training sample contains a task prompt, dialogue context formed from previous system and user utterances, and a target output text.The model is trained to generate the target output conditioned on the prompted dialogue context.
- Task Application: For intent classification, the model receives a prompted dialogue and generates the corresponding intent label text.The example maps a weather-query dialogue to the output label [get_weather].
- Learning Objective: Training uses a maximum-likelihood objective, and the same objective is applied when adapting the pre-trained model to a downstream task with labeled data.Multi-task pre-training optimizes the model over data annotated for different TOD tasks.
- Implementation: The reported model sizes are initialized from T5-small, T5-base, and T5-large, with approximately 60M, 220M, and 770M parameters, respectively.The models are pre-trained for 10 epochs with samples truncated to a maximum length of 1024.
4 Experiments
PPTOD is evaluated across end-to-end dialogue modelling, dialogue state tracking, and intent classification, including full-training and low-resource settings. It outperforms prior methods broadly, with especially strong gains when training data is scarce.
- Model design analysis: Plug-and-play models yield better results than cascaded counterparts, while using database state generally improves performance and can provide an around 4× speedup over MinTL.The plug-and-play model generates POL and NLG in parallel after predicting DST and retrieving the database state.
- End-to-end dialogue modelling: PPTOD performs better than previous state-of-the-art methods on seven of eight end-to-end metrics across MultiWOZ 2.0 and 2.1.It uses a single architecture without an additional language model for output re-ranking.
- End-to-end dialogue modelling: PPTOD consistently outperforms all compared baselines in low-resource end-to-end settings, with larger gains when fewer training samples are used.At 20% of the training data, it achieves scores comparable to systems trained on the full dataset.
- Dialogue state tracking: PPTODlarge obtains the highest accuracy among generation-based dialogue state tracking approaches on both MultiWOZ datasets.Classification-based methods remain stronger but rely on a fixed ontology, whereas PPTOD directly generates outputs.
- Dialogue state tracking: With only 1% of MultiWOZ 2.0 training data, PPTOD surpasses the strongest SOLOIST baseline by 18 points of accuracy.The low-resource DST evaluation varies training data from 1% to 20%.
- Intent classification: PPTOD is comparable with existing intent-classification methods, while PPTODlarge achieves the best results in low-resource-30 and full-training settings.The model requires no extra parameters for the classification task, and results average five runs with standard deviations.
5 Further Analysis
Further analyses compare plug-and-play with cascaded generation and examine dialogue multi-task pre-training and human judgments. Plug-and-play models reduce error accumulation and latency, while jointly using heterogeneous annotations improves broad performance and responses score higher on truthfulness and coherency.
- Plug-and-Play vs Cascaded Generation: Plug-and-play models yield better end-to-end results than cascaded counterparts, while database state generally improves both formulations.Database state provides additional grounding information.
- Plug-and-Play vs Cascaded Generation: With database state, the plug-and-play model achieves a better overall score than MinTL with around 4× speedup.The result suggests benefits in both generation accuracy and inference latency.
- Dialogue Multi-Task Pre-Training: Without pre-training, vanilla T5-small performs poorly in low-resource settings across all evaluated tasks.The analysis characterizes prior pre-training knowledge as indispensable for strong low-resource performance.
- Dialogue Multi-Task Pre-Training: Pre-training data annotated for an individual TOD task improves the corresponding downstream task, including DST, NLG, and end-to-end dialogue modelling.DST pre-training improves downstream DST, while NLG pre-training improves BLEU in end-to-end dialogue modelling.
- Dialogue Multi-Task Pre-Training: PPTODsmall achieves the best results on most evaluation metrics, indicating compatibility among differently annotated pre-training data.The joint utilization of all pre-training data supports the best overall performance.
- Human Evaluation: PPTOD significantly outperforms SOLOIST on truthfulness and coherency, while both systems perform comparably with the reference on fluency.The truthfulness and coherency differences are significant by Sign Test with p-value < 0.05; fluency is comparable with p-value > 0.4.
6 Conclusion
The conclusion presents PPTOD as a unified plug-and-play model for task-oriented dialogue and reports extensive automatic and human evaluations across benchmark tasks and resource settings. These evaluations show that PPTOD outperforms current state-of-the-art systems on various metrics.
- Conclusion: PPTOD unifies task-oriented dialogue understanding and response generation in a plug-and-play model.The model supports both functions within one unified system.
- Conclusion: A dialogue multi-task pre-training strategy augments PPTOD’s ability to complete task-oriented dialogue-related tasks.The conclusion identifies this strategy as an additional contribution.
- Conclusion: Automatic and human evaluations show that PPTOD outperforms current state-of-the-art systems across various evaluation metrics.Experiments cover three benchmark task types in high-resource and low-resource settings.
Ethical Statement
The ethical statement describes the task-oriented dialogue setting as user assistance without bias toward participants and reports no attached privacy or ethical issues for the previously published datasets used.
- Ethical Statement: The systems aim to interact with and assist users in fulfilling their goals without bias toward participants.This statement describes the intended interaction and assistance process.
- Ethical Statement: The datasets come from previously published works and are viewed by the authors as having no attached privacy or ethical issues.The statement also says the authors honor and support the ACL Code of Ethics.
A Dataset Details
The pre-training corpus combines heterogeneous human-written task-oriented dialogue and intent datasets spanning multiple domains and annotation types. The listed resources support NLU, DST, POL, and NLG-related training or evaluation.
- Dialogue Corpora: MetaLWOZ contains NLG annotations spanning over 47 domains for generating responses in unseen domains.It is designed to improve natural-language response generation in unseen domains.
- Intent Corpora: SNIPS, CLINC, and ATIS provide NLU annotations covering over 20, 150, and 22 intents respectively.Their domains include crowdsourced multi-domain utterances, out-of-scope intent detection, and airline travel information.
- Dialogue Corpora: KVRET contains DST and NLG annotations for calendar scheduling, weather information retrieval, and point-of-interest navigation.It is an in-car personal assistant dataset spanning three domains.
- Dialogue Corpora: WOZ and CamRest676 contain DST and NLG annotations for restaurant-domain dialogues collected with a Wizard-of-Oz procedure.Both datasets are described as restaurant-domain resources.
- Dialogue Corpora: Frames, TaskMaster, Schema-Guided, and MSR-E2E provide annotations for DST, POL, and NLG-related tasks.These datasets cover trip booking, multiple domains, 17 domains, and three domains respectively.
B Low-Resource MultiWOZ Evaluation
The MultiWOZ 2.0 low-resource evaluation reports results across five runs for each training-data setting. It presents complete results together with means and standard deviations.
- Each low-resource setting uses five runs with different training-data selections and random seeds.
- The evaluation reports complete results alongside the mean and standard deviation for each setting.
- The experiments are conducted on MultiWOZ 2.0 under different low-resource settings.
C Human Evaluation Guidelines
The human evaluation asks annotators to judge system responses on understanding, truthfulness, coherency, and fluency. Each dimension uses a three-level scale from 0 to 2 with task-specific criteria.
- Human evaluators assess responses for understanding, truthfulness, coherency, and fluency.The guidelines define separate criteria for each feature.
- Understanding: Understanding scores whether the response captures the user’s goal and intent, from 0 for no understanding to 2 for complete understanding.
- Truthfulness: Truthfulness scores whether response facts are supported by the reference, from contradiction at 0 to full support or inference at 2.
- Coherency: Coherency scores logical consistency with dialogue context, from complete irrelevance at 0 to logical coherence at 2.
C.4 Fluency
The fluency guideline evaluates grammatical correctness and readability using a three-level scale. The accompanying dialogue example illustrates a generated conversation involving restaurant and hotel requests.
- Fluency: Fluency is scored from 0 to 2 based on whether the response is grammatical and easy to understand.
- Fluency: A score of 2 denotes a grammatical, understandable response, while 1 permits minor errors and 0 indicates an unreadable response.
- Generated dialogue example: The PPTODbase example begins with a request for an expensive restaurant serving Indian food for dinner.
- Generated dialogue example: The example then covers restaurant location, hotel booking, unavailable requests, booking information, and co-referenced hotel information.