PaperlayerView exact revision on arXiv

Source-linked AI summary

UBAR: Towards Fully End-to-End Task-Oriented Dialog Systems with GPT-2

Yunyi Yang, Yunhao Li, Xiaojun Quan

arXiv:2012.03539v2cs.CL

TL;DR

Task-oriented dialog systems need to handle multi-domain conversations without the error propagation and ground-truth context assumptions of prior approaches. UBAR fine-tunes GPT-2 on complete dialog-session sequences and evaluates it using generated context. It achieves state-of-the-art results across multiple settings, while analyses show session-level modeling and generated context are central to its operation.

  • Problem

    Task-oriented dialog systems must support varied domains and real conversations, while pipeline cascades can propagate errors across subtasks.

  • Method

    UBAR fine-tunes GPT-2 on complete dialog-session sequences containing user utterances, belief states, database results, system acts, and responses, using generated context during evaluation.

  • Results

    UBAR achieves state-of-the-art performance across compared response generation, policy optimization, end-to-end modeling, and dialog state tracking settings.

  • Takeaways & Limitations

    Session-level training and generated dialog context are essential for UBAR to operate as a fully end-to-end task-oriented dialog system in realistic use.

Abstract

from arXiv · show

This paper presents our task-oriented dialog system UBAR which models task-oriented dialogs on a dialog session level. Specifically, UBAR is acquired by fine-tuning the large pre-trained unidirectional language model GPT-2 on the sequence of the entire dialog session which is composed of user utterance, belief state, database result, system act, and system response of every dialog turn. Additionally, UBAR is evaluated in a more realistic setting, where its dialog context has access to user utterances and all content it generated such as belief states, system acts, and system responses. Experimental results on the MultiWOZ datasets show that UBAR achieves state-of-the-art performances in multiple settings, improving the combined score of response generation, policy optimization, and end-to-end modeling by 4.7, 3.5, and 9.4 points respectively. Thorough analyses demonstrate that the session-level training sequence formulation and the generated dialog context are essential for UBAR to operate as a fully end-to-end task-oriented dialog system in real life. We also examine the transfer ability of UBAR to new domains with limited data and provide visualization and a case study to illustrate the advantages of UBAR in modeling on a dialog session level.

Introduction

Task-oriented dialog systems must support varied domains while remaining coherent and practical in real conversations. UBAR addresses pipeline error propagation by modeling entire dialog sessions with generated intermediate content.

  • Task-oriented dialog systems assist users with tasks such as hotel reservations and ticket booking through natural language conversations.
  • Wide domain coverage creates a need for stronger transfer ability and practicality in real conversations.
  • Traditional systems estimate belief states, query databases, select system acts, and generate responses through consecutive modules.
  • Pipeline error propagation across cascaded components can harm subsequent subtasks.
  • UBAR fine-tunes GPT-2 on entire dialog sessions containing user utterances, belief states, database results, system acts, and responses.
  • UBAR is evaluated with user utterances and all content it generated, rather than relying on ground-truth generated context.
  • On MultiWOZ, UBAR achieves state-of-the-art performance across response generation, policy optimization, end-to-end modeling, and dialog state tracking settings.

Related Work

Task-oriented dialog research has progressed toward generation and end-to-end modeling, with GPT-2 enabling unified treatment of dialog components. Prior systems still differ in supervision, decoder design, database handling, or pretraining requirements.

  • Research has moved from classification and modularized modeling toward generation and end-to-end task-oriented dialog modeling.
  • GPT-2-based approaches showed that task-oriented dialog information could be modeled in unified text sequences.
  • SimpleTOD incorporates database results and generates belief states and system acts, while SOLOIST uses out-of-domain pretraining and does not require system-act annotation.
  • UBAR extends GPT-2-based task-oriented dialog work toward fully end-to-end modeling over whole dialog sessions.

Method

UBAR fine-tunes GPT-2 to model a complete task-oriented dialog session as one sequence of user inputs and intermediate and final system outputs. Its context includes generated history, while preprocessing converts dialog components into structured text spans.

  • Modeling on a Dialog Session Level: Each turn follows user utterance, belief state, database result, system act, and delexicalized response generation.
  • Modeling on a Dialog Session Level: A complete T-turn dialog is represented as one sequence containing those five components for every turn.
  • Modeling on a Dialog Session Level: At turn t, UBAR conditions generation on current user input and all previous generated outputs, including belief states, database results, acts, and responses.
  • Modeling on a Dialog Session Level: Unlike turn-level methods with ground-truth response history, UBAR incorporates belief states, database results, and system acts in its dialog context.
  • Domain-Adaptive Pre-processing: Preprocessing delexicalizes responses, converts intermediate information into spans, and surrounds components with special start and end tokens.
  • Domain-Adaptive Pre-processing: Belief states encode domain-slot-value goals, database results indicate matched-entity counts, and system acts use decoupled domain and act spans.
  • Architecture and Training Objective: UBAR fine-tunes GPT-2, a unidirectional Transformer language model, on session-level task-oriented dialog data.
  • Architecture and Training Objective: UBAR uses the standard language-modeling objective of maximizing next-word prediction probability without additional objectives such as next-utterance classification.

Experiments

UBAR is evaluated on MultiWOZ 2.0 and 2.1 across response generation, policy optimization, end-to-end modeling, and dialog state tracking. It achieves state-of-the-art results across these settings using generated intermediate dialog content in end-to-end evaluation.

  • Datasets: MultiWOZ 2.0 contains 8,438 multi-domain dialogues across seven domains, while MultiWOZ 2.1 fixes noisy state annotations.Experiments use both dataset versions; each validation and test set contains 1,000 dialogues, excluding hospital and police.
  • Metrics: Inform measures correct entities, Success measures requested information answered, BLEU measures response fluency, and the combined score summarizes overall quality.The combined score is (Inform+Success)×0.5+BLEU; joint goal accuracy evaluates dialog state tracking.
  • Baselines: UBAR is compared with GPT-2-based SimpleTOD and SOLOIST alongside HDSA, SFN+RL, ARDM, and DAMD.A separate DST-UBAR comparison includes TRADE, DSTQA, DST-Picklist, and SST.
  • Response Generation: UBAR achieves the best response-generation inform rate, success rate, and combined score, although its BLEU score is slightly lower than HDSA.The results indicate stronger task-completion grounding than language-surface performance in this setting.
  • Policy Optimization: 3.5 points: UBAR improves SOLOIST’s policy-optimization combined score while achieving higher inform and success rates without additional pre-training data.The policy context uses ground-truth belief states and database results, followed by generated acts and responses.
  • End-to-End Modeling: Almost 10 points: UBAR lifts the end-to-end combined score and achieves state-of-the-art performance on all metrics.In this setting, UBAR generates the belief state, queries the database, then generates the system act and response using generated dialog context.

Analysis and Discussion

UBAR is analyzed through context ablations, session-level versus turn-level training, and transfer experiments. The results show that generated intermediate context supports efficient end-to-end operation, while transfer improves with limited new-domain data but remains data hungry.

  • Dialog Context: UBAR can operate with much shorter context than turn-level methods because belief states retain much of the user’s goal information.Using only the previous turn underperforms using all previous turns but still outperforms other state-of-the-art methods, and shorter context is more computationally efficient.
  • Dialog Context: Ground-truth belief states slightly improve results, whereas ground-truth acts and responses substantially reduce performance by misleading the model about what it has already generated.The realistic evaluation therefore uses generated context rather than ground-truth content.
  • Session-Level vs. Turn-Level: Belief states and system acts outperform user utterances and responses, as well as the turn-level URUR model, in end-to-end modeling.This supports the greater importance of intermediate dialog representations and the difficulty of inferring them from turn-level dialog history.
  • Domain Transfer: UBAR’s transfer experiments test zero-shot and few-shot end-to-end modeling by withholding one domain and training on the other four domains.The experiments exploit shared ontology across domains, while performance varies when a held-out domain contains more unique concepts.
  • Domain Transfer: 20 points: few-shot BM improves the combined score over zero-shot BM on the held-out domain when fine-tuned with 100 dialog sessions.Few-shot BM performance drops on the original domains, indicating catastrophic forgetting to some extent; the gap from full-data UBAR underscores end-to-end modeling’s data hunger.

Visualization and Case Study

The visualization and case study illustrate how UBAR updates belief states from prior beliefs and current user input, and why generated context can preserve coherent task progress. The examples connect this behavior to adaptive entity completion during later turns.

  • Attention Visualization: UBAR’s attention visualization examines how generated belief states attend to prior belief states and the current user utterance.The figure’s x-axis contains belief states, the previous response, and the current user utterance; the y-axis contains generated belief states.
  • Attention Visualization: UBAR copies consistent prior constraints, such as a train’s destination and departure, then attends to the user utterance to update the travel day.This supports maintaining and adjusting belief states without rereading all previous utterances and responses each turn.
  • Case Study: In the hotel case study, generated context lets UBAR provide a specific hotel entity after the user requests booking an entity omitted from the first response.The figure compares responses based on generated versus oracle context and highlights task-related entities.
  • Case Study: Ground-truth response context can make UBAR mistakenly believe that an important hotel entity was already provided, causing it to omit that entity later.The case explains why generated context can outperform ground-truth context for maintaining session coherence and task completion.

Conclusion

UBAR approaches end-to-end task-oriented dialog in a realistic session-level setting. It generates belief states, system acts, and responses from user utterances and its own generated content.

  • Conclusion: UBAR is trained and evaluated on dialog-session sequences containing user utterances and all content it generates.The generated content includes belief states, system acts, and responses.
  • Conclusion: The paper uses extensive experiments and analyses to demonstrate the reported advantages of session-level modeling and GPT-2 for end-to-end task-oriented dialog.
Loading 2012.03539v2…