PaperlayerView exact revision on arXiv

Source-linked AI summary

InstructDial: Improving Zero and Few-shot Generalization in Dialogue through Instruction Tuning

Prakhar Gupta, Cathy Jiao, Yi-Ting Yeh, Shikib Mehri, Maxine Eskenazi, Jeffrey P. Bigham

arXiv:2205.12673v2cs.CL

TL;DR

Dialogue instruction tuning has not been systematically explored despite dialogue spanning multiple task types. INSTRUCTDIAL organizes diverse dialogue datasets into a shared instruction-tuning framework and reports improved zero-shot and few-shot performance across dialogue tasks.

  • Problem

    Instruction tuning had not been systematically studied for dialogue tasks, which span language understanding, generation, and domain-specific interaction.

    Source: §1 · pp. 1–2

  • Method

    INSTRUCTDIAL converts 59 openly available dialogue datasets into 48 text-to-text tasks and adds meta-tasks intended to encourage models to follow instructions.

    Source: §1 · p. 2

  • Results

    Instruction tuning improves zero-shot and few-shot performance across varied dialogue tasks, though performance patterns differ across tasks and model sizes.

    Source: §1 · p. 2, §5.1.3 · p. 7

  • Takeaways & Limitations

    The framework establishes baseline and upper-bound performance and integrates task-specific metrics for dialogue tasks.

    Source: §1 · p. 2

    The study does not explore in-context few-shot learning through examples because prompt length can exceed models’ maximum input length.

    Source: §7 · p. 10

Abstract

from arXiv · show

Instruction tuning is an emergent paradigm in NLP wherein natural language instructions are leveraged with language models to induce zero-shot performance on unseen tasks. Instructions have been shown to enable good performance on unseen tasks and datasets in both large and small language models. Dialogue is an especially interesting area to explore instruction tuning because dialogue systems perform multiple kinds of tasks related to language (e.g., natural language understanding and generation, domain-specific interaction), yet instruction tuning has not been systematically explored for dialogue-related tasks. We introduce InstructDial, an instruction tuning framework for dialogue, which consists of a repository of 48 diverse dialogue tasks in a unified text-to-text format created from 59 openly available dialogue datasets. Next, we explore cross-task generalization ability on models tuned on InstructDial across diverse dialogue tasks. Our analysis reveals that InstructDial enables good zero-shot performance on unseen datasets and tasks such as dialogue evaluation and intent detection, and even better performance in a few-shot setting. To ensure that models adhere to instructions, we introduce novel meta-tasks. We establish benchmark zero-shot and few-shot performance of models trained using the proposed framework on multiple dialogue tasks.

1 Introduction

Instruction tuning uses natural-language instructions to support generalization to unseen tasks, but it has not been systematically studied for dialogue. INSTRUCTDIAL brings diverse dialogue tasks into a shared framework and tests cross-task performance while encouraging models to follow instructions.

1 Introduction

  • INSTRUCTDIAL collects 59 dialogue datasets into 48 tasks in a text-to-text framework spanning generation, classification, and evaluation across task-oriented and open-ended settings.The tasks are drawn from different domains.Source: §1 · p. 2
  • Prior instruction-tuning work focused mainly on general NLP, while existing dialogue coverage was limited and did not span tasks and domains well.The cited dialogue tasks were collected through crowdsourcing.Source: §1 · pp. 1–2
  • The framework investigates whether tuning across many dialogue tasks and instructions supports performance on unseen dialogue tasks.The targeted tasks include intent detection, response generation, and domain-specific emotion classification.Source: §1 · p. 2
  • To encourage instruction following, the authors vary task formulations for the same input context and add instruction-specific meta-tasks.One example asks models to select the instruction matching an input-output pair.Source: §1 · p. 2
  • The authors report improved zero-shot and few-shot performance across varied dialogue tasks and provide analyses, baselines, upper bounds, and task-specific metrics.They also identify sensitivity to instruction wording and task interference as areas for improvement.Source: §1 · p. 2

2 Related Work

Prior dialogue work used pretraining and multitask learning, while instruction-tuning research largely targeted general NLP. INSTRUCTDIAL applies instruction tuning across broader dialogue tasks, datasets, and domains.

Pre-training and Multi-Task learning in Dialogue

  • Dialogue pretraining and multitask learning have been used in both open-domain conversation and task-oriented dialogue, with reported few-shot benefits from multitask pretraining.INSTRUCTDIAL covers both settings and adds task definitions and constraints to multitask learning.Source: §2 · p. 2

Instruction Tuning

  • INSTRUCTDIAL organizes dialogue tasks into a taxonomy that includes classification, generation, and evaluation.Its figure distinguishes classification in green from generation in orange.Source: Figure 2 · p. 3
  • Instruction-tuning work collected broad general-NLP task sets, but dialogue-specific studies covered fewer domains and datasets than INSTRUCTDIAL.Prior dialogue work examined prompt-based few-shot learning without fine-tuning and task-specific instructions for task-oriented dialogue.Source: §2 · pp. 2–3

3 Methodology

INSTRUCTDIAL frames dialogue instruction tuning as cross-task learning with task information supplied through natural-language instructions and related metadata. It organizes diverse dialogue tasks in a shared sequence-to-sequence schema and adds meta-tasks and a none-of-the-above option.

3.1 Instruction Tuning Background

  • In cross-task evaluation, a model trained on one task is tested on an unseen task, with instructions and other task information guiding the expected output.The task information may include prompts, definitions, constraints, and examples.Source: §3.1 · p. 3

3.2 Task Collection

  • The task collection covers classification, generation, evaluation, editing, pretraining, safety, and specialized-domain tasks drawn from open-access dialogue datasets.Examples include response selection, response editing, toxicity detection, and advice-giving.Source: §3.2 · p. 3

3.3 Task Schema and Formatting

  • Each task instance uses a shared sequence-to-sequence schema with a task definition, instance inputs, constraints, prompt, and output.The authors manually compose 3–10 definitions and prompts per task, randomly selecting them during testing; input fields use special tokens.Source: §3.3 · pp. 3–4, Figure 3 · p. 4

3.4 Meta Tasks

  • Instruction selection and instruction-binary meta-tasks train models to associate instructions with input-output pairs.The binary task asks whether an instruction leads from a given input to a given output.Source: §3.4 · pp. 4–5

3.5 None-of-the-above Options

  • For 10% of training instances, classification tasks add “none of the above” as either the correct answer or a distractor.When it is correct, the gold label is removed from the options and NOTA becomes the output.Source: §3.5 · p. 5

4 Experimental Setup

The experiments fine-tune two encoder-decoder models that were already instruction-tuned on general NLP tasks. Training samples task definitions and prompts, with different hardware setups for the two models.

4.1 Model Details

  • The study fine-tunes T0-3B and BART0; DIAL-BART0 is the main experimental model, while DIAL-T0 has 3B parameters and is described as impractical on popular affordable GPUs.BART0 has 406 million parameters, eight times fewer than T0-3B.Source: §4.1 · p. 5

4.2 Training Details

  • Training samples at most 5,000 instances per task and randomly assigns each instance a task definition and prompt.Inputs are truncated to 1,024 tokens and targets to 256 tokens; DIAL-BART0 and DIAL-T0 use different two-GPU setups.Source: §4.2 · p. 5

5 Experiments and Results

The experiments test zero-shot transfer to unseen dialogue tasks, automatic response evaluation, and zero- and few-shot performance on intent prediction, slot filling, and dialogue state tracking. Across these settings, INSTRUCTDIAL-tuned models show transfer gains, with results also varying by task, model size, and training exposure.

5.1.3 Results and Discussion

  • INSTRUCTDIAL-tuned DIAL-BART0 and DIAL-T0 outperform their base models on all six unseen tasks, about tripling performance on Eval selection, Relation classification, and Begins with generation.They also outperform GPT-3 on all tasks except Dialfact classification.Source: §5.1.3 · p. 6, Table 1 · p. 6
  • Adding 100 instances per test task improves DIAL-BART0 by 12–16% on three selection/classification tasks and 30–50% on Begins with generation and Relation classification.With 5,000 instances per test task, DB-Full achieves high performance across all test tasks; its Dialfact and relation-classification results approach state of the art.Source: §5.1.3 · p. 7
  • Removing meta-tasks causes a large drop on unseen classification, while removing NOTA causes a slight drop; removing instructions worsens performance on every task.Replacing BART0 with Bart-large especially reduces Eval and Answer selection performance, supporting the authors’ conclusion that general-NLP pretraining helps dialogue instruction tuning.Source: §5.1.3 · p. 7
  • Unseen-task performance rises sharply through 20–25 training tasks and continues increasing more gradually as further tasks are added.Figure 4 summarizes accuracy for four classification/selection tasks and RougeL for two generation tasks.Source: §5.1.3 · p. 7, Figure 4 · p. 7

5.1.4 Analysis

  • DIAL-BART0’s scores vary by instruction wording, while common classification errors include outputs outside the allowed classes.Such invalid outputs occur for 2.5% of DIAL-BART0 inputs and 4.8% of DIAL-T0 inputs; some tested tasks remained difficult zero-shot but were quickly learned from a few instances.Source: §5.1.4 · pp. 7–8

5.2 Zero-shot Automatic Response Evaluation

  • On zero-shot response relevance evaluation, DIAL-T0 ranks first or second on most evaluation sets when compared with human ratings.The model is trained without evaluation tasks and is evaluated on DSTC-10 relevance ratings.Source: §5.2 · p. 8

5.3.1 Intent Prediction

  • With 10 examples per intent class, DIAL-BART0 achieves competitive few-shot BANKING77 results and outperforms PPTOD, which has about twice as many parameters.DIAL-BART0 also improves substantially over BART0 in the zero-shot setting.Source: §5.3.1 · p. 9

5.3.2 Slot Filling

  • On Restaurant8k, INSTRUCTDIAL yields a 36.9-point zero-shot slot-filling improvement; few-shot DSTC8 experiments also show significant gains over the baseline.The DSTC8 few-shot setup trains on 25% of the data across four domains.Source: §5.3.2 · p. 9, Table 4 · p. 9

6 Conclusion

InstructDial applies instruction tuning to dialogue tasks and adds meta-tasks intended to encourage models to follow instructions. Models show zero-shot performance on some unseen tasks and few-shot performance on dialogue tasks, but gains do not extend to every unseen task.

6 Conclusion

  • InstructDial is an instruction-tuning framework for dialogue containing multiple tasks created from openly available dialogue datasets; it also introduces two meta-tasks intended to encourage the model to attend to instructions.Source: §6 · p. 9
  • Two instruction-specific meta-tasks are proposed to encourage models to pay attention to instructions.Source: §6 · p. 9
  • Models trained on InstructDial show good zero-shot performance on unseen tasks and good few-shot performance on dialogue tasks.The paper gives dialogue evaluation as a zero-shot example and intent prediction and slot filling as few-shot examples.Source: §6 · p. 9
  • Instruction tuning does not benefit every unseen test task, and the paper identifies instruction wording invariance and task interference as areas for improvement.Source: §6 · p. 10

7 Limitations

The framework and its findings have several stated scope limits: instructions are limited in number and English-only, zero-shot gains vary by task, and the work excludes in-context few-shot learning and non-text modalities.

7 Limitations

  • Instructions and prompts are limited in number and English-only, and the authors suggest crowdsourcing or automatic augmentation for greater quantity and language diversity.Source: §7 · p. 10
  • Zero-shot improvement is not significant on some tasks, including relation classification and infilling missing utterances, and performance on many tasks remains below few-shot and full-shot performance.Source: §7 · p. 10
  • The study does not explore in-context few-shot learning because long prompts can exceed models’ maximum input length, or composition of multiple tasks through instructions.Source: §7 · p. 10
  • InstructDial includes only text-based tasks; incorporating vision and audio datasets is left to future work.Source: §7 · p. 10

8 Ethics and Broader Impact

InstructDial is presented as a way to add and evaluate multiple dialogue-system functionalities, while making sub-billion-parameter models accessible to researchers. The authors also note potential dataset bias and risks from easier-to-train zero- and few-shot methods.

8 Ethics and Broader Impact

  • The framework allows task-oriented and domain-specific dialogue tasks to be added and evaluated, supporting diagnosis across different tasks and potentially richer user interactions.Source: §8 · p. 10
  • The framework allows training models below one billion parameters, making them more accessible to the research community.Source: §8 · p. 10
  • Biases in the datasets may propagate to unseen tasks, and easier-to-train zero- and few-shot methods may increase both model benefits and risks.Source: §8 · p. 10
  • The framework includes safety-related datasets and tasks for toxic-response classification and recovery-response generation, and an empathy-related task using EmpatheticDialogues.Source: §8 · p. 10

A Additional implementation details

The appendix specifies data sampling, training, and decoding choices, and gives an example of multiple instruction-guided tasks applied to one conversation.

A Additional implementation details

  • Training data is capped at 5,000 instances per task, with a randomly assigned task definition and prompt for each instance.Inputs are truncated to 1,024 tokens and target outputs to 256 tokens.Source: A Additional implementation details · p. 17
  • Both models train for three epochs with a 5e-5 learning rate and Adam; DIAL-BART0 and DIAL-T0 use different GPUs and batch sizes, with effective batch size 72 for each.Main Table 1 results are reported over three runs, each using a new training-data sample.Source: A Additional implementation details · pp. 17–18
  • Classification uses greedy decoding, while generation uses top-p sampling with p = 0.7 and temperature 0.7, with repetition penalty 1.2.Source: A Additional implementation details · p. 18

B Sample conversation and Instructions

C Datasets used in tasks

Table 9 lists the datasets used for each task.

D Configuration of experiments

The section describes experiment training configurations and provides examples of the task instructions, inputs, and datasets used.

  • Each experiment configuration identifies its base model and the tasks used to train it.Source: D Configuration of experiments · p. 18, Table 10 · p. 22
  • A sample conversation shows task-specific instructions and prompts producing outputs for multiple tasks on the same input.Source: Table 7 · p. 19
  • A task-sample table pairs general and specific task types with example inputs formatted using a task definition and prompt.It also explains abbreviations including [CTX], [Q], [RES], [EOT], and [EOD].Source: Table 8 · p. 20
  • A task-and-dataset table maps each specific task to the datasets used for it.Source: Table 9 · p. 21

Related papers

Loading 2205.12673v2…