PaperlayerView exact revision on arXiv

Source-linked AI summary

Simulated Chats for Building Dialog Systems: Learning to Generate Conversations from Instructions

Biswesh Mohapatra, Gaurav Pandey, Danish Contractor, Sachindra Joshi

arXiv:2010.10216v4cs.CLcs.AI

TL;DR

Dialog datasets can be costly to create, and training data are scarce for many tasks. This paper trains instruction-conditioned user and agent simulators on a small share of crowd conversations to generate more dialogs. The generated data improve low-resource results on MultiWOZ 2.0 and PersonaChat, though they do not improve models already trained on all human-generated data.

  • Problem

    Many dialog tasks lack sufficient training data, while creating large crowd-sourced datasets can be time-consuming and expensive.

    Source: §1 · p. 1

  • Method

    The paper trains instruction-conditioned user and agent simulators on a subset of crowd-sourced conversations to generate dialogs conditioned on instructions and a knowledge base.

    Source: §1 · p. 2

  • Results

    7-13% improvement in MultiWOZ 2.0 combined score and 2-10% improvement in Persona Chat Hits@1 are reported in low-resource settings.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    The simulation framework is described as applicable to diverse dialog tasks, including task-oriented and persona-based chat, without strict domain or dataset assumptions.

    Source: §5 · p. 9

    The technique does not improve performance when added to models trained on 100% human-generated data.

    Source: §4.6.1 · p. 8

Abstract

from arXiv · show

Popular dialog datasets such as MultiWOZ are created by providing crowd workers an instruction, expressed in natural language, that describes the task to be accomplished. Crowd workers play the role of a user and an agent to generate dialogs to accomplish tasks involving booking restaurant tables, calling a taxi etc. In this paper, we present a data creation strategy that uses the pre-trained language model, GPT2, to simulate the interaction between crowd workers by creating a user bot and an agent bot. We train the simulators using a smaller percentage of actual crowd-generated conversations and their corresponding instructions. We demonstrate that by using the simulated data, we achieve significant improvements in low-resource settings on two publicly available datasets - the MultiWOZ dataset and the Persona chat dataset.

1 Introduction

Crowd-sourced dialog creation is costly, while many dialog tasks lack sufficient training data. The paper simulates crowd-worker interactions with instruction-conditioned user and agent bots, then uses their conversations to improve low-resource dialog models.

1 Introduction

  • Dialog training data are scarce for most tasks, and creating large crowd-sourced datasets can be time-consuming and expensive.Crowd workers follow natural-language task instructions and interact as user and agent; task-oriented agents may query a knowledge base.Source: §1 · p. 1
  • The framework uses user and agent simulators to generate conversations conditioned on instructions and a knowledge base, trained with 5-20% of crowd-sourced conversations.The user simulator receives instructions; the agent simulator uses a knowledge base and maps dialog context to a belief state for retrieval.Source: §1 · p. 2
  • The approach generates task-oriented and persona-guided conversations, and the authors propose applying it to diverse dialog systems through different forms of instructions.The paper also describes the generated conversations as training data for final end-task models.Source: §1 · p. 2
  • 7-13% improvement in MultiWOZ 2.0 combined score and 2-10% improvement in Persona Chat Hits@1 are reported for low-resource settings; generated conversations were also judged grammatically sound and meaningfully progressive.The paper reports these findings alongside a human study of simulated dialogs.Source: §1 · p. 2

2 Related Work

Prior simulator approaches include template-based dialog generation and methods using human input to add variation. This paper positions its approach as more general and less cumbersome, and as a dialog-data application of teacher models.

2 Related Work

  • Template-based simulators generate dialogs with hand-crafted templates, while another simulator approach uses human involvement to add variation.The authors contrast their approach with both lines of work.Source: §2 · p. 2
  • The authors describe their approach as more general and less cumbersome than template-based dialog generation, and say it needs no human involvement beyond providing diverse instructions.They support the generality claim by demonstrating the approach on two different tasks.Source: §2 · p. 2
  • The paper applies teacher-model data generation to dialog systems, which the authors say had not previously been done to their knowledge.They distinguish this use from prior teacher-student work on knowledge distillation, dynamic loss functions, and meta-learning adaptation.Source: §2 · p. 3

3 Model

The framework trains user and agent bots on a subset of original conversations, then uses them to generate dialogs conditioned on task instructions and, for task-oriented dialogs, a knowledge base. Each bot generates candidate utterances and selects among them; the agent additionally generates and executes belief-state queries.

3 Model

  • The framework trains GPT-2 teacher bots on 5% or 20% of the original data, then mixes generated dialogs with the low-resource data to train student models.Task-oriented dialogs also use a belief-state generator, and Longformer selectors choose among generated responses.Source: §3 · p. 3

3.1 Overview

  • Dialog generation models user and agent turns conditioned on instructions and a knowledge base, with the first turn starting from empty dialog history.The user bot uses instructions and history; the agent bot uses history, the latest user utterance, and the knowledge base.Source: §3.1 · p. 3

3.2 User Bot

  • The user bot generates candidate utterances from instructions and dialog history, then selects the highest-probability candidate as the next user turn.Its GPT-2 generator produces delexicalised utterances, which are lexicalised using instruction values.Source: §3.2 · p. 4, §3.2.1 · p. 4

3.2.2 Response Selector

  • Nucleus sampling supplies diverse user-response candidates, while a Longformer selector scores context-response relevance and is trained with positive and negative examples using triplet loss.The negative samples include random responses, responses already in the context, and concatenated responses; the margin is α = .05.Source: §3.2.1 · p. 4, §3.2.2 · p. 4

3.3 Agent Bot

  • The agent bot generates a belief-state query, retrieves matching knowledge-base entities, and selects a response from candidates conditioned on the context and query.The interaction continues until the user bot emits an end-of-dialogue token; corresponding user and agent modules do not share weights.Source: §3.3 · pp. 4–5

3.3.1 Belief State (query) Generator

  • The belief-state generator uses GPT-2 to produce a sequence of domain and key-value tokens from dialog history and the latest user utterance, then executes the sampled state against the knowledge base.Belief states are sampled by greedy decoding.Source: §3.3.1 · p. 5

3.3.2 Response Generator

  • The agent response generator receives context, the latest user utterance, the belief state, and the number of matching knowledge-base entities, then lexicalises sampled candidates with belief-state values.The model uses the entity count rather than the full entities, since entity-specific information can be filled during lexicalisation.Source: §3.3.2 · p. 5

3.3.3 Response Selector

  • A Longformer response selector scores each agent candidate using the context, latest user utterance, and response; generated utterances, belief states, and knowledge-base results form the dialog.Figure 4 illustrates candidate responses generated using greedy and nucleus sampling, with the highest-scoring response selected.Source: §3.3.3 · p. 5, Figure 4 · p. 5

4 Experiments

The experiments test simulated-dialog augmentation across task-oriented and persona-based conversations, comparing low-resource training with larger-data settings and established baselines. Simulated data improves low- and medium-resource results, while the human study finds strong fluency and grammar but slightly lower instruction relevance than crowd-generated dialogs.

4.1 Datasets

  • MultiWOZ has 8,438 training conversations across seven domains, while PersonaChat contains 10,907 dialogs tied to 1,155 distinct character profiles.MultiWOZ conversations include task instructions; PersonaChat speakers condition dialogue on profiles.Source: §4.1.1 · p. 6, §4.1.2 · p. 6

4.2 Data Generation using Simulators

  • At 5% and 20% training-data levels, simulators use instructions from the remaining MultiWOZ data to generate conversations; PersonaChat uses one bot conditioned on each speaker’s persona.For MultiWOZ, generated conversations are added to the original data; at 100%, simulated conversations are generated from the same training instructions and the resulting set doubles in size.Source: §4.2 · p. 6

4.3 End-Task Models

  • End-task evaluation uses GPT-2-small-based Soloist and T5-small-based MinTL-T5 on MultiWOZ, and a GPT-2-small model on PersonaChat.Soloist is initialized only with GPT-2-small for this study; the goal is to test simulated-data augmentation.Source: §4.3 · p. 6

4.4 Baselines

  • Baselines include non-augmented end-task models, DAMD, dialog-state and paraphrase augmentation methods, and Lost in Conversation for PersonaChat.The MultiWOZ comparisons also include T5-based paraphrase augmentation; PersonaChat compares GPT-2-small with and without augmented data.Source: §4.4 · pp. 6–7

4.5 Metrics

  • MultiWOZ evaluation uses BLEU, Inform, Success, and Combined scores, while PersonaChat uses Hits@1 and Perplexity.Combined is BLEU + 0.5 × (Inform + Success); the authors note that delexicalization differences make direct model comparisons inaccurate.Source: §4.5 · p. 7

4.6.1 Use of simulated data in End-task

  • MultiWOZ combined scores rose 7–8% with 5% data and 7% for MinTL and 13% for Soloist with 20% data; PersonaChat Hits@1 gains were 2–10%.The MultiWOZ table reports BLEU, Inform, Success, and Combined scores; the PersonaChat table reports performance across dataset sizes.Source: Table 1 · p. 7, Table 2 · p. 7, §4.6.1 · p. 8
  • Adding simulated data to 20%-trained models improved combined scores but remained below 100%-human-data results; augmentation did not improve models trained on all human data.Soloist rose from 69.5 to 78.3 versus 88.6 with 100% human data; MinTL rose from 72 to 76.6 but also remained below its 100%-data result.Source: §4.6.1 · p. 8

4.6.2 Human Study

  • In a blind study, six participants rated instruction-matched crowd and simulated dialogs; simulated dialogs scored well on fluency and grammar but were slightly less relevant to instructions.The authors attribute the relevance difference to simulated conversations sometimes omitting information from instructions, which also helps explain why augmentation did not improve models trained on all human data.Source: §4.6.2 · pp. 8–9, Table 3 · p. 8

5 Conclusion

The framework generates meaningful conversations that improve end-task dialog-model performance in low-resource settings. The authors describe it as applicable to task-oriented and persona-based dialog, and outline further appendix materials.

5 Conclusion

  • 7-13% performance improvement in low-resource MultiWOZ 2.0 settings and 2-10% increase in PersonaChat Hits@1 are reported with additional simulated data.The framework is described as applicable to task-oriented and persona-based dialog.Source: §5 · p. 9
  • The framework does not make strict assumptions about the domain or dataset, and is applied to task-oriented dialog and persona-based chat.The authors suggest comparing augmentation methods and how they may be combined in future work.Source: §5 · p. 9

A Appendix Overview

  • The appendix overview points readers to hyperparameter settings, cost analysis, evaluation inconsistencies, paraphrase-based augmentation, further experiments, and an additional qualitative study.These topics are assigned to Sections B through H.Source: A Appendix Overview · p. 11

B Hyperparameter Settings

The experiments use GPT2-small response generators and Longformers as response selectors; MultiWOZ additionally uses belief-state models, while PersonaChat uses a single user model.

B Hyperparameter Settings

  • MultiWOZ training uses 5%, 10%, 30%, or 100% of the training data, with a learning rate of 1e-5 and Adam.The response generators use GPT2-small; response selectors use Longformers.Source: B Hyperparameter Settings · p. 11
  • MultiWOZ uses separate user and agent bots, each with query models, response generators, and response selectors.GPT2-small has 12 layers, 768 hidden size, and 117M parameters; the Longformers have 12 layers, 1024 hidden size, and 149M parameters.Source: B Hyperparameter Settings · p. 11
  • PersonaChat has no belief-state generator and trains only a single user model, using GPT2-small generators, Longformer selectors, and Adam at a 1e-5 learning rate.Source: B Hyperparameter Settings · p. 11

C Cost comparison

The authors compare the cost and time of simulated MultiWOZ data creation with crowd sourcing and report substantially lower estimated cost for their method.

C Cost comparison

  • Under $400 is estimated for simulated data creation, compared with up to $7.5k for crowd sourcing 10.4K dialogs.The simulated-data estimate uses an Amazon EC2 P3 instance at $3.06 per hour; the crowd estimate assumes two workers per conversation, one hour per conversation, and a $6 minimum wage.Source: C Cost comparison · p. 11
  • Five models generate the dialogues in one day each on a single V100 GPU for the 100%-data setting.The crowd-sourced MultiWOZ process required 1249 workers.Source: C Cost comparison · p. 11
  • The cost comparison estimates that 1249 workers generating 10.4K dialogs would produce 15-16 dialogues per worker.The estimate assumes two workers per conversation and one hour to generate each conversation.Source: C Cost comparison · p. 11

D Evaluation Inconsistency

Differences in delexicalization across models affect MultiWOZ evaluation because the official script scores delexicalized agent responses using domain-specific requestable tokens.

D Evaluation Inconsistency

  • PARG, DAMD, and MinTL use delexicalization that differs from the original MultiWOZ code.Source: D Evaluation Inconsistency · p. 11
  • Different delexicalization schemes created discrepancies in MultiWOZ model evaluation because the official script checks requestable-token domains.For example, the passage contrasts [train_address] for a Train utterance with [hotel_address], and notes that generic [value_address] tokens reduce dialog complexity and can raise inform and success scores.Source: D Evaluation Inconsistency · p. 12
  • Soloist and MinTL use the delexicalization scheme from the original MultiWOZ code, leading to a small drop in their overall performance.Source: D Evaluation Inconsistency · p. 12

E Soloist Paraphrase Model

The Soloist paraphrase model augments low-resource training data with a mixture of T5-generated and PARG paraphrases, bringing the augmented data to the original dataset size.

F MultiWoz 2.1 Performance

On MultiWOZ 2.1, simulated data improves Soloist's combined metric in low-to-medium-resource settings. The paper also illustrates generation under small instruction changes.

  • The MultiWOZ 2.1 comparison uses Soloist with oracle and generated belief states; few baselines report results on this revised dataset version.Source: F MultiWoz 2.1 Performance · p. 12
  • 18–25% improvement in combined metrics is reported for low-to-medium-resource settings using simulated data on MultiWOZ 2.1.Source: F MultiWoz 2.1 Performance · p. 12
  • Figure 5 examples compare conversations generated from original and slightly perturbed MultiWOZ instructions, illustrating the generator's reported robustness.Source: Figure 5 · p. 12

G Single Goal Dialogs

The section evaluates combining simulated single-goal dialogues across domains and tests their use on conversations with multiple goals. It also examines generation from perturbed instructions.

G Single Goal Dialogs

H Qualitative Study - Instruction Perturbation

Loading 2010.10216v4…