PaperlayerView exact revision on arXiv

Source-linked AI summary

Language Models as Few-Shot Learner for Task-Oriented Dialogue Systems

Andrea Madotto, Zihan Liu, Zhaojiang Lin, Pascale Fung

arXiv:2008.06239v2cs.CLcs.LG

TL;DR

Task-oriented dialogue learning is costly because its modules require separate training data, and few-shot transfer-learning approaches still fine-tune task-specific parameters. This paper evaluates GPT-2 priming across four tasks; priming shows potential in most, though DST remains a substantial gap and input length and computation constrain the approach.

  • Problem

    Task-oriented dialogue modules require training data that can be laborious and expensive to collect, while fine-tuned few-shot approaches require task-specific parameters.

    Source: §1 · pp. 1–2

  • Method

    The paper evaluates few-shot GPT-2 priming across NLU, DST, ACT, and NLG, using example prefixes without updating model parameters.

    Source: §1 · p. 2

  • Results

    Priming shows promising results in NLU, ACT, and NLG, sometimes matching or exceeding the weakest fine-tuning baseline, while DST retains a large baseline gap.

    Source: §5 · p. 4

  • Takeaways & Limitations

    The experiments demonstrate potential for LM priming across common task-oriented dialogue tasks, although performance is not uniform across tasks.

    Source: §6 · p. 5

    GPT-2’s 1024-token input limit restricts most tasks to 15 or fewer shots, limiting comparability with models using more examples.

    Source: §5 · p. 4

Abstract

from arXiv · show

Task-oriented dialogue systems use four connected modules, namely, Natural Language Understanding (NLU), a Dialogue State Tracking (DST), Dialogue Policy (DP) and Natural Language Generation (NLG). A research challenge is to learn each module with the least amount of samples (i.e., few-shots) given the high cost related to the data collection. The most common and effective technique to solve this problem is transfer learning, where large language models, either pre-trained on text or task-specific data, are fine-tuned on the few samples. These methods require fine-tuning steps and a set of parameters for each task. Differently, language models, such as GPT-2 (Radford et al., 2019) and GPT-3 (Brown et al., 2020), allow few-shot learning by priming the model with few examples. In this paper, we evaluate the priming few-shot ability of language models in the NLU, DST, DP and NLG tasks. Importantly, we highlight the current limitations of this approach, and we discuss the possible implication for future work.

1 Introduction

Task-oriented dialogue systems depend on separately trained NLU, DST, dialogue-policy, and NLG modules, while collecting training data can be laborious and expensive. The paper evaluates whether priming GPT-2 with examples can support few-shot learning across these tasks without task-specific parameter updates.

1 Introduction

  • Training four separate dialogue modules can require costly data collection, including expert annotation for dialogue policy.The paper identifies few-shot models as important given this collection burden.Source: §1 · p. 1
  • Prior few-shot approaches include fine-tuned TOD-BERT for NLU, DST, and DP, and task-specific methods for slot filling and NLG.Related work also includes template rewriting and meta-learning approaches.Source: §1 · pp. 1–2
  • The paper evaluates GPT-2 priming across NLU, DST, DP, and NLG without updating parameters, using SMALL (117M), LARGE (762M), and XL (1.54B) models.The authors motivate priming as allowing one model to perform multiple tasks without task-specific parameter sets.Source: §1 · p. 2

2 Basic Notation and Tasks

The paper introduces four task-oriented dialogue tasks through their inputs and outputs. It treats dialogue as alternating utterances from a user and system, and defines few-shot learning as generalizing from a small number of input-output pairs.

2 Basic Notation and Tasks

  • A dialogue is an alternation of utterances between two speakers, and few-shot learning supplies a small number of input-output pairs for generalization.Source: §2 · p. 2
  • NLU extracts slot-value pairs and an intent from a user utterance, with slot filling producing a dictionary and intent classification predicting a class.Unmentioned slots may take the value None; intent classification can sometimes include domain classification.Source: §2 · p. 2
  • DST predicts dialogue-level slot-value pairs and can update the prior turn’s state using the latest input.The described common approach uses the previously generated state and updates required slots.Source: §2 · p. 2
  • ACT predicts the next speech act from dialogue state, but this paper simplifies policy learning to multi-label dialogue-act classification.The input is a user utterance classified into a set of possible speech acts.Source: §2 · p. 2
  • NLG maps a dialogue act and its slot-value dictionary to a natural-language utterance.Source: §2 · p. 2

3 Priming the LM for few-shot learning

The method primes a language model with fixed few-shot prefixes and a test input, then uses generated tokens as predictions. It uses binary, value-based, and generative prefixes for different task types.

3 Priming the LM for few-shot learning

  • The priming setup places few-shot examples and the input to predict in a prefix that remains fixed during inference, with generated tokens serving as the prediction.The paper illustrates this setup for intent recognition in Figure 1.Source: §3 · p. 2
  • Binary prefixes handle classification, treating even multiclass problems as binary and requiring one prefix and forward pass per class.The paper’s Figure 3 gives an ACT-task example.Source: §3 · p. 3, Figure 3 · p. 3
  • Value-based prefixes assign a slot’s value from an utterance, including None when no value is provided, and are repeated for each slot.The Figure 2 caption identifies a slot-filling example; the procedure requires forwarding the model for each slot.Source: §3 · p. 3, Figure 2 · p. 3
  • Generative prefixes instruct the model to produce natural language from source information such as an NLG input.Source: §3 · p. 3

4 Experiments and Results

The experiments evaluate LM priming on NLU, DST, ACT, and NLG using task-specific datasets, metrics, shot limits, and fine-tuned baselines. The maximum number of examples varies because tasks fit differently within GPT-2’s 1024-token input limit.

4 Experiments and Results

  • Shot counts vary by task because different numbers of examples fit within GPT-2’s 1024-token maximum input.Source: §4 · p. 3
  • SNIPS experiments evaluate slot filling with CoNLL F1 and intent classification with accuracy, using value-based and binary prefixes with maxima of 15 and 10 shots, respectively.The intent baseline is RoBERTa fine-tuned on 10 samples; detailed results are referenced in Tables 1–2 and Figure 5.Source: §4 · p. 3
  • MultiWoZ DST uses only the latest user utterance as input, updates predictions across turns, and reports joint and slot accuracy.Baselines are TOD-BERT and BERT fine-tuned on 10% of the data, equivalent to 500 examples; LM priming uses at most 15 shots.Source: §4 · pp. 3–4
  • MultiWoZ ACT uses only the system utterance, evaluates F1-score, and compares a binary-prefix model using at most 15 shots with BERT and TOD-BERT fine-tuned on 500 examples.Unlike Wu et al. (2020), the input excludes dialogue history and the user utterance; Figure 3 illustrates the task.Source: §4 · p. 4
  • FewShotWOZ NLG uses BLEU and slot error rate, comparing a generative-prefix model with SC-LSTM, GPT-2, and SC-GPT-2 fine-tuned on 50 examples.The LM prefix permits at most 20 shots because of limited context; the caption identifies the systems as from Peng et al. (2020b).Source: §4 · p. 4, Figure 4 · p. 4

5 Analysis and Limitation

LM priming shows promise against the weakest fine-tuning baseline in NLU, ACT, and NLG, but a substantial DST gap remains. Results vary with model size, and the approach is constrained by inference demands and GPT-2’s context length.

5 Analysis and Limitation

  • Larger models perform better in NLU and NLG, while GPT-2 LARGE outperforms XL in DST and ACT.The authors say further investigation is needed to determine whether changing prefixes could improve larger models.Source: §5 · p. 4
  • LM priming matches or exceeds the weakest fine-tuning baseline in NLU, ACT, and NLG, but still has a large gap in DST.The fine-tuning baseline uses more shots.Source: §5 · p. 4
  • Binary and value-based generation requires a forward pass for each class or slot, while early generative-format experiments performed poorly.The authors are unsure whether larger LMs such as GPT-3 would perform better in the generative format.Source: §5 · p. 4
  • GPT-2’s 1024-token input limit restricts most tasks to no more than 15 shots, making comparisons with models using more shots difficult.Source: §5 · p. 4

6 Conclusion

The paper evaluates LM-priming few-shot learning across NLU, DST, ACT, and NLG. It reports that larger LMs often perform better and sometimes match or exceed the weakest fine-tuning baseline, while noting computational cost and limited context size.

6 Conclusion

  • The experiments demonstrate the potential of LM-priming few-shot learning across NLU, DST, ACT, and NLG.Source: §6 · p. 5
  • Larger LMs are better few-shot learners in most tasks, and sometimes achieve results similar to or better than the weakest fine-tuning baseline.Source: §6 · p. 5
  • The paper identifies computational cost and limited word-context size as limitations, and proposes future benchmarks of dialogue-specific and longer-context LMs, adversarial triggers, and end-to-end dialogue tasks.Source: §6 · p. 5

A Appendices

The appendices provide task-specific priming examples and results figures, alongside tables reporting task evaluation metrics.

A Appendices

Loading 2008.06239v2…