PaperlayerView exact revision on arXiv

Source-linked AI summary

Few-Shot Bot: Prompt-Based Learning for Dialogue Systems

Andrea Madotto, Zhaojiang Lin, Genta Indra Winata, Pascale Fung

arXiv:2110.08118v1cs.CLcs.AI

TL;DR

Conversational AI lacks a formal benchmark for prompt-based few-shot learning. The paper benchmarks language models across dialogue tasks and proposes a prompt-based skill selector and chatbot. Results suggest the approach is promising, with larger language models performing better.

  • Problem

    The paper identifies a lack of formal benchmarks for prompt-based few-shot learning in conversational AI, where dataset annotation costs are high.

    Source: §1 · p. 3

  • Method

    The paper benchmarks different-sized language models on dialogue tasks and proposes a training-free prompt-based classifier to select a skill from dialogue history.

    Source: §1 · p. 5, §6 · p. 25

  • Results

    Across dialogue tasks, prompt-based few-shot learning is promising, and larger language models perform better.

    Source: §6 · p. 25

  • Takeaways & Limitations

    New dialogue skills can be added through prompts without an expensive training process.

    Source: §1 · p. 5

    For most language models, the 2,048-token context limit constrains the number of shots, especially when examples require thousands of tokens.

    Source: §4.1 · p. 22

Abstract

from arXiv · show

Learning to converse using only a few examples is a great challenge in conversational AI. The current best conversational models, which are either good chit-chatters (e.g., BlenderBot) or goal-oriented systems (e.g., MinTL), are language models (LMs) fine-tuned on large conversational datasets. Training these models is expensive, both in terms of computational resources and time, and it is hard to keep them up to date with new conversational skills. A simple yet unexplored solution is prompt-based few-shot learning (Brown et al. 2020) which does not require gradient-based fine-tuning but instead uses a few examples in the LM context as the only source of learning. In this paper, we explore prompt-based few-shot learning in dialogue tasks. We benchmark LMs of different sizes in nine response generation tasks, which include four knowledge-grounded tasks, a task-oriented generations task, three open-chat tasks, and controlled stylistic generation, and five conversational parsing tasks, which include dialogue state tracking, graph path generation, persona information extraction, document retrieval, and internet query generation. The current largest released LM (GPT-J-6B) using prompt-based few-shot learning, and thus requiring no training, achieves competitive performance to fully trained state-of-the-art models. Moreover, we propose a novel prompt-based few-shot classifier, that also does not require any fine-tuning, to select the most appropriate prompt given a dialogue history. Finally, by combining the power of prompt-based few-shot learning and a Skill Selector, we create an end-to-end chatbot named the Few-Shot Bot (FSB), which automatically selects the most appropriate conversational skill, queries different knowledge bases or the internet, and uses the retrieved knowledge to generate a human-like response, all using only few dialogue examples per skill.

1. Introduction

Prompt-based few-shot learning offers a training-free approach to dialogue tasks, where the paper addresses a stated lack of formal benchmarks and introduces a prompt selector and Few-Shot Bot. The introduction presents broad task coverage and reports promising results, including some improvements over fully trained state-of-the-art models.

1. Introduction

  • Results show a promising trend across tasks, with improvements over full-shot state-of-the-art models in some cases.The benchmark covers 11 dialogue datasets.Source: §1 · p. 3
  • The paper benchmarks prompt-based few-shot learning across 11 datasets and 15 conversational tasks, using examples in the LM context without training or fine-tuning.Tasks include chit-chat, knowledge-grounded response generation, dialogue state tracking, and document retrieval.Source: §1 · pp. 3, 5
  • The proposed Skill-Selector maps dialogue histories to the most likely prompt or model and requires no training, allowing new skills to be added without an expensive training process.The paper frames this as a meta-task and describes a perplexity-based classifier.Source: §1 · pp. 4–5
  • The Few-Shot Bot combines prompt-based skills to select dialogue capabilities, access knowledge sources, track user preferences, and generate responses without fine-tuning.The listed sources include the Internet, Wikipedia, and knowledge graphs.Source: §1 · p. 5

2. Methodology

The methodology uses prompted language models to map dialogue histories to responses or task-specific outputs, and combines few-shot prompting with a selector and retrievers in the Few-Shot Bot.

2.1 Prompt-Based Few-Shot Learning

  • Few-shot prompting conditions a language model on textual input-output examples and a task input to generate the target output.The method uses no model updates during prompting.Source: §2.1 · pp. 5–6

2.2 Prompt-Based Learning vs. Fine-Tuning

  • Prompt-based learning leaves model weights unchanged, unlike fine-tuning, and the study considers GPT-2, GPT-Neo, and GPT-J models of different sizes.The listed models range from 0.1B to 6B parameters.Source: §2.2 · p. 6, §2.3 · p. 6

2.4 Datasets

  • The benchmark covers response generation, conversational parsing, and controlled-style generation across the listed dialogue datasets, with task-specific automatic metrics.The response-generation tasks include open-chat, task-oriented, and knowledge-grounded settings; parsing includes dialogue state tracking, retrieval, and query or path generation.Source: §2.4 · pp. 6–7, §2.5 · p. 7

2.6 Skill Selector

  • The Skill-Selector treats skill choice as classification from a dialogue history to a task or dataset label.The labeled dialogue set is split into training, validation, and test data to model P(Y|X).Source: §2.6 · pp. 7–8

2.6.1 Prompt-Based Few-Shot Skill Selector

  • The prompt-based selector chooses the prompt with the highest conditional probability for a dialogue history, avoiding a training step but requiring one LM forward pass per skill.The forward-pass cost grows linearly with the number of skills and can be expensive, though batching is possible.Source: §2.6.1 · pp. 8–9

2.7 The Few-Shot Bot (FSB)

  • The Few-Shot Bot combines one-shot skill prompts with language models and retrievers to select skills, access knowledge sources, recall prior conversations, and generate responses.Its components include a language model, a prompt per skill, and knowledge retrievers; parsing skills use Wikipedia, web search, or a knowledge graph.Source: §2.7 · p. 9, Language Models · p. 9

2.8 Interactive Algorithm

  • At each turn, the bot updates dialogue history, selects a skill, retrieves knowledge when needed, updates memory, and generates a response from the selected prompt.The process repeats across turns.Source: §2.8 · pp. 9–10
  • Skills can be added, removed, or edited by changing their prompts without model training, but the authors describe the bot as a proof of concept rather than deployment-ready.Source: §2.8 · p. 10

3. Results

The experiments benchmark prompt-based few-shot methods across dialogue tasks, with shot counts constrained by each language model’s input capacity. GPT-J (6B) consistently outperforms smaller models, while more shots do not always improve performance; conversational parsing remains harder than response generation.

3.1 Experiments Details

  • Experiments report means and standard deviations over three runs using validation-sampled prompts, without tuning prompt templates on validation data.Generation uses greedy decoding with a maximum length of 150; beam search is omitted partly because beam 8 requires 40GB of GPU RAM for a 6B half-precision model.Source: §3.1 · p. 10, Conversational Parsing · p. 12

Conversational Parsing

  • GPT-J (6B) consistently outperforms smaller models across tasks, with performance correlating linearly with model size.The reported size relationship is even stronger for conversational parsing; the authors suggest larger models or better prompts could eventually close its performance gap.Source: Conversational Parsing · pp. 11–12
  • More shots usually improve results, but one-shot prompts outperform many-shot prompts on knowledge-grounded WoW, WiT, and DialKG.The number of tested shots is constrained by model input length and dialogue turns and metadata.Source: §3.1 · p. 10, Conversational Parsing · p. 11
  • Conversational parsing is harder than response generation: FSB trails the state of the art by 24.5 RPrec. on WoW-Parse and reaches 13.9 JGA on single MWoZ-DST.The passage also notes a stronger parsing performance correlation with model size, but its following comparison is truncated.Source: Conversational Parsing · pp. 11–12
  • The prompt-based Skill-Selector performs comparably to fine-tuned RoBERTa base and large baselines without requiring fine-tuning.The reported advantage is that it is highly extendable.Source: Conversational Parsing · p. 12

3.3 Response Generation

Across response-generation tasks, Few-Shot Bot (FSB) uses few-shot prompts with varying results against fully trained systems. Performance depends on the dataset, shot count, model size, and—in image-grounded dialogue—an image-captioning step.

3.3.2 Empathetic Dialogues (ED)

  • The 6B FSB is competitive with some fully trained models, but results vary substantially by dataset and baseline.It performs comparably to AdapterBot, BST, and MBST on Persona Chat and Empathetic Dialogues, while trailing models including DDD on several reported comparisons.Source: §3.3.1 · p. 12, §3.3.2 · p. 13

3.3.3 Wizard of Wikipedia (WoW)

  • One-shot 6B FSB performs best on Wizard of Wikipedia, but it trails DDD and AdapterBot when both use gold Wikipedia knowledge.Source: §3.3.3 · p. 13

3.3.4 Image Chat (IC)

  • On Image Chat, the 6B FSB improves as shots increase and beats BST and DialoGPT, while matching MBST on F1 and BLEU-4.Compared with DDD, it has lower Rouge-L and BLEU-4 and higher perplexity, but comparable F1.Source: §3.3.4 · pp. 13–14

3.3.5 Wizard of Internet (WIT)

  • On Wizard of Internet, one-shot 6B FSB performs best, but its F1 and KF1 are much lower than all full-shot baselines.Source: §3.3.5 · p. 14

3.3.6 Controlled Generation (CG-IC)

  • For controlled Image Chat style generation, larger models and more shots improve performance, though comparisons with full-dialogue systems are not equivalent because FSB predicts only the last turn.Source: §3.3.6 · pp. 14–15

3.3.7 Multi-Session Chat (MSC)

  • In Multi-Session Chat, larger models correlate with better performance, more shots do not help, and FSB has slightly higher perplexity than fully trained BST.Source: §3.3.7 · p. 15

3.3.8 DailyDialogue (DD)

  • On DailyDialogue, the 6B FSB with one or six shots has better perplexity than DDD and NS+MR, but one point lower F1 than DDD.Source: §3.3.8 · p. 16

3.3.9 Stanford Multidomain Dialogue (SMD)

  • On SMD, eight-shot FSB has the highest weather-domain entity-F1 among compared systems, but scores lower in the other domains.For DialKG, more shots improve Rouge-L and BLEU but increase FeQA variance; FSB is competitive with some models on FeQA and Rouge-L, trails on BLEU, and remains behind NPH overall.Source: §3.3.9 · p. 16, §3.3.10 · p. 17

3.4 Conversational Parsing

The parsing tasks use dialogue histories to generate text queries, graph paths, persona information, or dialogue-state updates. Few-shot performance generally improves with larger models and more shots, but comparisons range from competitive to substantially behind existing systems.

3.4 Conversational Parsing

  • Conversational parsing maps dialogue histories to domain-specific queries for knowledge access, search, or dialogue-state tracking.Source: §3.4 · p. 17

3.4.1 WoW-Parse

  • FSB generates Wikipedia document titles from dialogue histories for retrieval; larger models and more shots improve performance, but FSB remains behind GENRE and ahead of DPR.GENRE uses constrained beam search; FSB is evaluated with retrieval precision.Source: §3.4.1 · p. 17

3.4.2 WiT-Parse

  • For WiT-Parse, FSB generates search-engine queries, including a None token when no query is needed; larger models and more shots improve performance, especially BLEU-4.The authors lack other baselines, and evaluate with response-generation metrics because running Bing retrieval is too expensive.Source: §3.4.2 · p. 18

3.4.3 MSC-Parse

  • In MSC-Parse, FSB generates user persona information for use in later conversations; larger models and more shots improve overall performance, with no other baselines reported.Results are averaged across the dataset's four sessions.Source: §3.4.3 · pp. 18–19

3.4.4 MWoZ-Parse

  • MWoZ-Parse generates dialogue-state updates from dialogue history using predicted rather than gold states; performance rises with model size and shot count, but reaches only 13.9% JGA.Evaluation uses single-domain dialogue and reports joint goal accuracy and slot accuracy.Source: §3.4.4 · p. 19

3.4.5 DialKG-Parse

  • DialKG-Parse generates graph paths by predicting tuple subject and relation, then retrieving and perplexity-ranking connected nodes; FSB beats simple baselines with one or two shots but trails current state of the art.The process repeats until a termination token; performance and model size correlate positively in the one-shot setting.Source: Figure 5 · p. 18, §3.4.5 · pp. 19–20

3.5 Skill Selection

Skill selection maps dialogue histories to the most likely prompt or skill. The prompt-based selector works without fine-tuning and is compared with fine-tuned RoBERTa.

3.5 Skill Selection

  • With one shot, the prompt-based Skill-Selector has 6% lower accuracy than RoBERTa fine-tuning, while RoBERTa base with six shots has the highest accuracy and F1.RoBERTa base's six-shot result has considerably higher standard deviation across three runs; larger models generally perform better, while additional shots help less.Source: §3.5 · p. 20

3.6 Ethical Considerations

The safety evaluation finds comparable offensive-language generation results to BST, but concerning responses to offensive language.

3.6 Ethical Considerations

  • FSB has alarmingly high positive affirmation and negative detection in offensive-language tests, and relatively high offensive response versus BST, motivating caution and safety layers before deployment.Source: §3.6 · p. 21

4. Limitation & Future Work

The paper identifies limits involving contamination uncertainty, the number of usable shots, access to larger models, and the absence of human evaluation. It also outlines dialogue tasks and datasets for future study.

4.1 Limitation

  • The experiments cannot rule out test samples appearing in LM pretraining, and the authors leave a more thorough contamination examination to future work.Verification is difficult when LM training data are unavailable and very large.Source: §4.1 · p. 21
  • FSB implements only four dialogue skills: PersonaChat, MSC, ED, and DD.Source: §4.1 · p. 22
  • A 2048-token input limit restricts shots, especially when structured or unstructured examples require thousands of tokens each.The authors suggest alternatives including improving task descriptions, prompt or adapter tuning, cloze-style questions, and automatically learning discrete prompts.Source: §4.1 · pp. 22–23
  • Although GPT-J (6B) achieved competitive performance to full-shot models, larger-model experiments were constrained by access to GPT-3 and GPT-Jurassic.The paper reports a long waiting list for GPT-3 access and a 10K-token-per-day limit for GPT-Jurassic.Source: §4.1 · p. 23

4.2 Human Evaluation

  • Only automatic evaluation was conducted, leaving response-generation and end-to-end chatbot results less reliable without a human study.The authors call for interactive human evaluation of FSB against systems such as Meena and BST.Source: §4.2 · p. 23

4.3 Future Work

  • Future work could expand coverage to additional task-oriented, multimodal, continual-learning, and open-domain dialogue datasets.The paper notes that its MultiWOZ evaluation uses a single domain and lists further dialogue tasks and datasets to explore.Source: §4.3 · pp. 23–24

5. Related Work

Prior prompt-based learning work spans many NLP and dialogue tasks, but dialogue studies have often focused on goal-oriented tasks or used gradient-based fine-tuning. This work situates its contribution as a broad study of gradient-free prompting across task-oriented dialogue and open-domain chit-chat.

Related work on prompt-based learning

  • Prompt-based learning offers few-shot performance without fine-tuning, but faces shot limits, instability, and prompt-engineering challenges.The paper notes approaches including calibration, prompt-order learning, automatic prompt selection, and embedding-based prompts as partial responses.Source: §5 · p. 24

Prompt-Based Learning in Dialogue Systems

  • Earlier dialogue prompt studies include fine-tuned NLU and NLG methods and gradient-free work on NLU, DST, and NLG.The paper characterizes existing dialogue prompt-learning studies as focused on goal-oriented tasks.Source: §5 · p. 24
  • Concurrent work on grounded response generation used training-based or continuous prompts requiring gradient updates, unlike this paper’s prompt-only approach.The compared work includes Zheng and Huang (2021), while the paper describes its own scope as task-oriented dialogue and open-domain chit-chat.Source: §5 · pp. 24–25

6. Conclusion

The paper benchmarks prompt-based few-shot learning across dialogue generation and parsing tasks, proposes a no-fine-tuning prompt selector, and builds FSB to select skills and retrieve knowledge. Its results suggest promise for dialogue systems, with larger LMs performing better.

6. Conclusion

  • Prompt-based few-shot learning is promising across dialogue tasks, and larger LMs perform better; FSB combines prompt selection with skill selection and knowledge retrieval to generate responses.The benchmark covers nine response-generation tasks and five conversational-parsing tasks; FSB uses only a few examples per skill.Source: §6 · p. 25

Appendix A.

The appendix provides prompt templates and examples for response-generation and conversational-parsing tasks. Its overview notes that DailyDialogue and Image Chat prompts predict all turns in the human-to-human dialogues.

Appendix A.

MSC-parse

This material catalogs model-size/metric figures across dialogue datasets alongside prompt examples and procedural pseudocode for parsing and controlled generation.

MSC-parse

Loading 2110.08118v1…