PaperlayerView exact revision on arXiv

Source-linked AI summary

A Simple Language Model for Task-Oriented Dialogue

Ehsan Hosseini-Asl, Bryan McCann, Chien-Sheng Wu, Semih Yavuz, Richard Socher

arXiv:2005.00796v4cs.CL

TL;DR

Task-oriented dialogue is traditionally split into independently trained modules, creating modular dependencies that can propagate errors. SimpleTOD recasts all sub-tasks as one causal language-modeling sequence, achieving state-of-the-art results across dialogue state tracking and end-to-end action and response generation on MultiWOZ.

  • Problem

    Traditional task-oriented dialogue systems train separate modules for understanding, dialogue management, and response generation, whose dependencies can propagate errors.

  • Method

    SimpleTOD uses one causal language model trained with multi-task maximum likelihood on a unified sequence covering dialogue state tracking, action decisions, and response generation.

  • Results

    SimpleTOD outperformed prior methods in dialogue state tracking and end-to-end action and response generation, achieving 55.76 joint goal accuracy on MultiWOZ.

  • Takeaways & Limitations

    Pre-trained open-domain causal language models such as GPT-2 can transfer language understanding to a unified task-oriented dialogue system.

  • Takeaways & Limitations

    Generated belief states can conflict with database information, so the best result ignores database search results entirely.

Abstract

from arXiv · show

Task-oriented dialogue is often decomposed into three tasks: understanding user input, deciding actions, and generating a response. While such decomposition might suggest a dedicated model for each sub-task, we find a simple, unified approach leads to state-of-the-art performance on the MultiWOZ dataset. SimpleTOD is a simple approach to task-oriented dialogue that uses a single, causal language model trained on all sub-tasks recast as a single sequence prediction problem. This allows SimpleTOD to fully leverage transfer learning from pre-trained, open domain, causal language models such as GPT-2. SimpleTOD improves over the prior state-of-the-art in joint goal accuracy for dialogue state tracking, and our analysis reveals robustness to noisy annotations in this setting. SimpleTOD also improves the main metrics used to evaluate action decisions and response generation in an end-to-end setting: inform rate by 8.1 points, success rate by 9.7 points, and combined score by 7.2 points.

1 Introduction

Task-oriented dialogue traditionally separates understanding, decision-making, and response generation into pipeline components, but SimpleTOD recasts all sub-tasks as one causal language-modeling problem. On MultiWOZ, it achieves state-of-the-art tracking and improves end-to-end action and response metrics, while analyses examine robustness and design choices.

  • Motivation: Traditional task-oriented dialogue systems train NLU, dialogue management, and NLG components independently, creating dependencies that can propagate errors.Many systems also pass belief states between modules rather than using the entire dialogue history at every turn.
  • Approach: SimpleTOD recasts all task-oriented dialogue sub-tasks as a single causal language-modeling task trained end to end.The approach is designed to model dependencies among the sub-tasks while leveraging pre-trained causal language models such as GPT-2.
  • Results: 55.76 joint goal accuracy on MultiWOZ surpasses prior work for dialogue state tracking.The paper also reports robustness to noisy-labeled annotations in dialogue state tracking.
  • Results: +8.1 inform rate and +9.7 success rate over prior work are reported when belief states and action decisions are generated rather than retrieved from an oracle.This setting tests a full task-oriented dialogue system with end-to-end action and response generation.
  • Analysis: Ablations identify user/system and endof(segment) tokens and pre-training as important design choices.Larger SimpleTOD versions are not always better for end-to-end MultiWOZ, and the project provides noisy-annotation discoveries, a cleaned test set, and code.

2 Related Work

Prior task-oriented dialogue research commonly isolates individual modules, while newer work connects selected components or pursues end-to-end generation. SimpleTOD differs by using a simpler language-model-based formulation that outperforms previous models on dialogue state tracking and end-to-end evaluation.

  • Task-oriented dialogue: Prior work often targets one task, including intent detection, dialogue state tracking, policy prediction, or response generation.These approaches typically evaluate a specific module rather than the complete task-oriented dialogue system.
  • Bridging sub-tasks: Some approaches connect multiple sub-tasks, but may rely on oracle dialogue states or omit dialogue state tracking evaluation.Joint action-response generation and GPT-2 response generation are examples of these partial integrations.
  • End-to-end dialogue: End-to-end approaches aim to reduce pipeline error propagation, human effort, and task-specific design by generating or retrieving responses from dialogue history and knowledge bases.Several such systems optimize neural encoder-decoder models without modular supervision.
  • Pre-trained language models: GPT-2-based prior systems explored response generation or alternating user-system roles, but did not evaluate all sub-tasks in the end-to-end setting.Other work fine-tuned GPT-2 on MultiWOZ and reported lower dialogue-state-tracking and end-to-end performance than earlier models.
  • SimpleTOD: SimpleTOD uses a simpler input sequence, embedding design, training algorithm, and pre-training setup than previous language-model-based approaches.The paper reports that this simpler model outperformed previous models on dialogue state tracking and end-to-end evaluation.

3 Methods

SimpleTOD frames task-oriented dialogue as a single causal sequence-generation problem, conditioning action and response generation on dialogue context, belief state, and database results. It uses a Transformer-based causal language model trained on concatenated task sequences and evaluates the approach on MultiWOZ.

  • Task-Oriented Dialogue: SimpleTOD generates belief states, actions, and delexicalized responses within one end-to-end sequence-generation framework.The training sequence concatenates context, belief state, database results, actions, and response.
  • Task-Oriented Dialogue: At turn t, the model reads all previous turns, predicts a belief state, queries a database, and conditions action prediction on the resulting information.Database results include the number of matching rows and, depending on the setting, booking status information.
  • Causal Language Modeling: The model trains on concatenated sequences [Ct; Bt; Dt; At; St] by minimizing autoregressive negative log-likelihood.The language-model objective factorizes sequence probability with the chain rule and predicts each next token from preceding tokens.
  • Architecture: A causal Transformer uses multi-head attention with a mask that prevents attending to future tokens, followed by feedforward, normalization, residual, and output-scoring layers.During training, output scores enter cross-entropy loss; during generation, final-token scores are normalized with softmax for the next-token distribution.
  • Implementation: SimpleTOD uses pretrained DistilGPT2 tokenization and weights, truncates sequences longer than 1024 tokens, and receives database results during inference.The database input may be excluded during training because experiments indicate that doing so can be advantageous.
  • Evaluation Details: Experiments use MultiWOZ, a multi-domain human-human dialogue dataset, and evaluate belief-state tracking, action and response generation, fluency, and a combined score.MultiWOZ contains 10,438 multi-turn dialogues across seven domains, with police and hospital excluded from evaluation.

4 Experimental Results and Discussion

SimpleTOD achieves state-of-the-art results across dialogue state tracking and fully end-to-end action and response generation on MultiWOZ, using a single causal model without extra supervision or annotation cleaning. Its performance remains strong across database-search settings, although generated belief states can conflict with database information.

  • Overall performance: SimpleTOD achieves state-of-the-art dialogue state tracking and end-to-end action and response generation on MultiWOZ.The end-to-end setting requires generating belief states, actions, and responses rather than using oracle intermediate outputs.
  • Dialogue state tracking: 55.76 joint goal accuracy surpasses prior work for dialogue state tracking on MultiWOZ.Compared methods include TRADE, DSTQA, DST-Picklist, SST, and TripPy.
  • Robustness and evaluation conditions: SimpleTOD achieves the best performance without test-label cleaning or normalization, using raw original annotations.Several comparison systems apply label cleaning, format normalization, typo correction, or label-variation handling.
  • Database-search settings: SimpleTOD outperforms prior work with oracle, ignored, and dynamically computed database-search results.The experiments include directly comparable oracle settings, no-database settings, and dynamic database matching.
  • Database-search settings: Database information can lower performance when generated belief states conflict with database entries.The paper gives restaurant-name discrepancies such as “pizza hut fenditton” versus “pizza hut fen ditton” as an example.

5 Analysis and Further Discussion

Further analysis examines special tokens, pre-training, noisy annotations, decoding, and long contexts. The results indicate that token delimiters and pre-training matter, while SimpleTOD remains effective with greedy decoding and noisy or extended dialogue context.

  • Special tokens: Performance on all metrics drops without <endof(segment)> and <user/system> tokens.End tokens also prevent excessively long belief-state, action, and response generations, while user/system distinctions are especially important.
  • Pre-training: Pre-training is important because single-sequence prediction lets SimpleTOD leverage language understanding learned from open-domain data.The analysis attributes this advantage to initializing the model with pre-trained weights.
  • Noisy annotations: SimpleTOD can track dialogue state and generate correct outputs despite noisy labels in MultiWOZ annotations.The analysis identifies four primary noisy-label types, including insufficient context, missing or unsupported labels, and misspellings.
  • Decoding: Greedy decoding is sufficient for SimpleTOD, while nucleus sampling degraded performance.The model does not require more costly beam-search, diverse-beam, or top-k strategies.
  • Long contexts: SimpleTOD accurately tracks dialogue state over multiple turns and long contexts, sometimes correcting earlier errors as more context becomes available.The paper treats this as further analysis rather than a primary contribution.

6 Conclusion

SimpleTOD uses one causal language model to treat dialogue state tracking, action decisions, and response generation as a single sequence prediction problem. On MultiWOZ, it outperformed prior methods across these tasks, remained effective over long context and noisy annotations, and required only greedy decoding.

  • SimpleTOD treats dialogue state tracking, action decisions, and response generation as one sequence for a single causal language model.The model leverages pre-trained language understanding from open-domain settings.
  • SimpleTOD outperformed all prior methods in dialogue state tracking and end-to-end action and response generation on MultiWOZ.
  • Pre-trained weights were essential, while special tokens marked user, system, and sub-task sequence segments.
  • SimpleTOD tracked dialogue state over long contexts with many turns and achieved new state-of-the-art results using greedy decoding despite noisy annotations.

7 Broader Impact

The work may simplify conversational agents, potentially improving efficiency in predictable communications. Broader deployment could also dehumanize automatable communication or frustrate users when automated solutions do not fit.

  • Simplified conversational agents could automate predictable communications and reduce time spent navigating APIs, webpages, and telephone systems.
  • Broader deployment might dehumanize automatable communication and frustrate users when predicted solutions do not apply.

A Input Representation and Method Overview

SimpleTOD concatenates dialogue context, belief states, database results, action decisions, and system responses into one training sequence. During inference, it pauses after belief-state generation to query a database, then resumes generation and lexicalizes the response.

  • Training concatenates context, belief states, database results, action decisions, and system responses into one sequence.The joint sequence is optimized with negative likelihood.
  • The model predicts each next token from the output state associated with the preceding input token.
  • Inference pauses after generating belief states, queries a database, and appends summarized results before generation resumes.
  • The resumed generation produces a delexicalized response that is lexicalized using slots and values from database results.

B SimpleTOD with Oracle information

Oracle-information experiments provide additional comparisons but are not end-to-end evaluations of SimpleTOD. They show a disconnect between oracle and practical settings, while MultiWOZ 2.1 results follow the same patterns as earlier evaluations.

  • These oracle settings are not end-to-end, because SimpleTOD is designed to generate the relevant information rather than receive it as oracle input.
  • Other methods can outperform SimpleTOD with oracle information, whereas SimpleTOD excels when oracle information is unavailable.
  • Oracle-information experiments evaluate action and response generation under settings using oracle belief states and, respectively, oracle or generated actions.
  • The experiments include comparable settings with oracle database-search results and settings that ignore those results.
  • MultiWOZ 2.1 evaluation results follow the same patterns as the earlier evaluation and support future comparisons on the improved dataset version.

C Dialogue State Tracking Analysis

SimpleTOD’s DST analysis examines long, multi-domain contexts and robustness to noisy annotations, alongside full end-to-end dialogue examples. It shows that the model can infer related cross-domain states and often recover correct belief states from incorrect annotations.

  • Understanding Long, Multi-domain Context: SimpleTOD tracks dialogue states across long contexts spanning attraction, hotel, and taxi domains.Taxi departure and destination are inferred from information in the attraction and hotel domains.
  • Regarding Noisy Annotations: The analysis reports robustness to noisy annotations across four defined noise types.The model can often generate the correct belief state when the annotation is incorrect.
  • Full Dialogue Example: Tables 15 and 16 present delexicalized and lexicalized full-dialogue outputs from the strongest SimpleTOD model.This setting requires generating all outputs while ignoring database-search results.
  • Regarding Noisy Annotations: The noisy-annotation examples include insufficient context, omitted belief states, unsupported labels, and misspelled belief-state values.These cases are illustrated in Tables 10–14.
Loading 2005.00796v4…