PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback

Jing Xu, Megan Ung, Mojtaba Komeili, Kushal Arora, Y-Lan Boureau, Jason Weston

arXiv:2208.03270v2cs.CLcs.AI

TL;DR

Human-model conversations can differ from the human-human data used to train dialogue agents, and such distribution shifts can reduce performance. The paper collects deployment conversations and feedback, then compares learning methods for using it. It reports that modular feedback, especially combined with binary feedback using DIRECTOR, yields its best results.

  • Problem

    Human-model conversations differ from the human-human training data, and current techniques can lose performance because they are not robust to such deviations.

    Source: §1 · p. 1

  • Method

    The authors deploy internet-augmented dialogue models, collect human conversations and varied feedback, and compare supervised learning, rejection sampling, model guiding, and reward-based learning.

    Source: §1 · pp. 1–2

  • Results

    Modular feedback outperforms feedback about only final responses, and combining modular and binary feedback with DIRECTOR yields the best results reported.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The authors report that iterative retraining brings further gains and that feedback collected with smaller models can improve much larger models.

    Source: §6 · p. 9

    Experiments used English-speaking U.S. Mechanical Turk workers, unlike organic public users, whose feedback and engagement tradeoffs may differ.

    Source: §7 · p. 9

Abstract

from arXiv · show

Frozen models trained to mimic static datasets can never improve their performance. Models that can employ internet-retrieval for up-to-date information and obtain feedback from humans during deployment provide the promise of both adapting to new information, and improving their performance. In this work we study how to improve internet-driven conversational skills in such a learning framework. We collect deployment data, which we make publicly available, of human interactions, and collect various types of human feedback -- including binary quality measurements, free-form text feedback, and fine-grained reasons for failure. We then study various algorithms for improving from such feedback, including standard supervised learning, rejection sampling, model-guiding and reward-based learning, in order to make recommendations on which type of feedback and algorithms work best. We find the recently introduced Director model (Arora et al., '22) shows significant improvements over other existing approaches.

1 Introduction

The paper studies learning from deployment conversations to address performance loss when human-model dialogue differs from the human-human data used for training. It collects varied feedback and compares learning methods, finding the strongest results when modular and binary feedback are combined with DIRECTOR.

1 Introduction

  • The study deploys internet-augmented dialogue models with crowdworkers and collects binary, free-form, and fine-grained feedback, including failure types and improvement suggestions.Workers first specify topics and tasks they want the chatbot to handle.Source: §1 · pp. 1–2
  • Modular feedback about errors in individual model components outperforms feedback about only the final response.Textual and binary feedback are also useful, but less so than modular feedback.Source: §1 · p. 2
  • With binary feedback, DIRECTOR works better than reranking or reward-based learning.The paper identifies DIRECTOR as a model-guiding method.Source: §1 · p. 2
  • Combining modular and binary feedback with DIRECTOR provides the best results reported.The paper also reports that continual learning improves results further and that feedback collected with 3B models can improve 175B models.Source: §1 · p. 2
  • The authors make the collected data and feedback, models, and code publicly available.The work compares supervised learning, rejection sampling, model guiding, and reward-based learning.Source: §1 · p. 2

2 Related Work

Prior work collects feedback from human-model conversations in symmetric and asymmetric settings, and also improves language skills from deployment in other domains.

2 Related Work

  • In symmetric human-model conversations, models can learn by mimicking human conversationalists, while asymmetric dialogue requires other forms of supervision.The LIGHT dialogue game illustrates the symmetric setting (Shuster et al., 2020); textual feedback and reward learning are among the approaches discussed for asymmetric conversations (Li et al., 2016a; Hancock et al., 2019).Source: §2 · p. 2
  • Related work studies learning from answers, textual feedback, and reward signals, as well as web-based response ranking.Examples include learning to ask questions from answers (Li et al., 2016b), textual feedback on low-quality responses (Li et al., 2016a), and stack ranking (Ouyang et al., 2022; Bai et al., 2022).Source: §2 · p. 2
  • Deployment learning also appears in language-data learning, web search, and broader NLP, while Nakano et al. (2021) study internet-augmented question answering rather than multi-turn dialogue.The cited examples include never-ending learning, web search, and Dynabench (Kiela et al., 2021).Source: §2 · p. 2

3 Deploying and Collecting Feedback

The study elicits user-desired chatbot skills, deploys conversational models to crowdworkers, and gathers feedback during task-oriented dialogues. It evaluates several model families and learning rounds using conversational ratings, a large collected dataset, and a held-out set of new skills.

3.1 Open-domain internet-driven skills

  • Researchers collected 1,108 task types via 152 annotators, covering diverse topics and activities; many require internet use or are made simpler by searching for entities or topics and conditioning responses on pertinent results.Annotators were asked to provide a topic, three related tasks, and descriptions of how they would assess task completion.Source: §3.1 · pp. 2–3

3.2 Conversing with models and receiving feedback

  • Crowdworkers chose tasks and conversed with a model for up to four turns, giving binary satisfaction labels, free-form feedback, or fine-grained improvement suggestions.When feedback supplied a better query, search context, or response, the system used it to continue the conversation and gather feedback on later stages.Source: §3.2 · pp. 3–4

3.3 Deployed Models

  • The deployed systems included non-internet BlenderBot (BB1), internet-enabled BlenderBot 2.0 (BB2) and SeeKeR, and 175B OPT and BlenderBot 3 models.BB2 uses search with a fusion-in-decoder approach, while SeeKeR generates a knowledge response before its final dialogue response.Source: §3.3 · p. 4

3.4 Evaluation

  • Performance was evaluated using the number of good responses, average final ratings, and breakdowns of error types.Error categories included better-search-query, results, or other issues.Source: §3.4 · p. 4

3.5 Collected Data

  • The FITS dataset contains over 210,000 utterances across more than 14,000 dialogues, with v1 and v2 deployment rounds and an unseen-skill test split.Three annotators judged task completion in 90.4% of conversations; the unseen split tests transfer to new topics and tasks.Source: §3.5 · pp. 4–5, Table 2 · p. 5

4 Feedback Learning Methods

The paper compares supervised response and module learning, text-feedback fine-tuning, reranking, reward-based training, and DIRECTOR as ways to learn from collected human feedback.

4.1 Supervised Learning of Responses

  • Supervised response learning fine-tunes on user-provided gold responses and can also include bot responses marked good, with validation data choosing their weighting.The authors describe a provided gold response as an easy-to-use, strong learning signal.Source: §4.1 · p. 5

4.2 Fine-grained Module supervision

  • Module supervision fine-tunes search-query generation on gold queries and trains on gold knowledge responses, using special tokens for BB2's knowledge-response targets.SeeKeR can directly fine-tune on gold knowledge responses; BB2 uses a separate supervised target because knowledge is implicit in FiD.Source: §4.2 · p. 5

4.3 Free-form Textual Feedback

  • Free-form textual feedback can be used as a fine-tuning target, with special tokens marking the feedback task in the input and target.This approach follows Hancock et al. (2019).Source: §4.3 · p. 5

4.4 Rejection Sampling/Reranking

  • A reward model trained on binary satisfaction labels can rerank a dialogue model's generated candidates by predicted satisfaction.The reward model is a 311M-parameter transformer trained with a classification loss on the concatenated context and response.Source: §4.4 · p. 6

4.5 Reward-based Learning

  • Reward-based learning generates candidates, selects the highest-ranked ones with a reward model, and fine-tunes the generation model on them.Unlike reranking alone, this procedure trains the generation model itself.Source: §4.5 · p. 6

4.6 Model-guiding with DIRECTOR

  • DIRECTOR jointly trains a language-model head and a task-suitability classifier on positive and negative examples, then combines their probabilities during decoding.The classifier is intended to steer generation away from undesirable sequences; in this setting, examples come from human feedback.Source: §4.6 · p. 6, Figure 2 · p. 6
  • DIRECTOR combines language-model and classifier probabilities at each left-to-right generation step to incorporate negative feedback into generation.Its two heads are trained jointly; the classifier head uses negative data while positive generations train both heads.Source: §4.6 · p. 6, Figure 2 · p. 6

5 Experimental Results

Internet augmentation and human feedback improve the tested dialogue models, with the strongest results from combining modular and binary feedback in DIRECTOR. Iterative retraining and feedback collected with smaller models also benefit later or larger models.

Internet-augmentation helps

  • Internet-augmented BB2 and SeeKeR outperform non-augmented BB1 in Good responses, and SeeKeR also markedly outperforms BB2.The reported rates are 24.8% for BB1, 33.2% for BB2, and 49.3% for SeeKeR.Source: §5 · pp. 6–7

Modular superior to non-modular feedback

  • Modular feedback about search queries or selected documents beats final-response-only supervision, improving Good responses by 1.7 points for BB2 and 4.5 points for SeeKeR.BB2 rises from 40.3% to 42.0%, while SeeKeR rises from 52.2% to 56.7%.Source: §5 · p. 7

Feedback methods

  • Free-form feedback improves both base models but trails supervised feedback by 3% for BB2 and 1% for SeeKeR; binary-feedback methods beat baselines, with DIRECTOR exceeding supervised SeeKeR by 3.3% in Good responses.The authors report broad feedback-related gains across human and automatic metrics, though not in every case.Source: §5 · p. 7

Combining multiple feedback signals

  • 47.0% Good responses for BB2 with modular-plus-binary DIRECTOR feedback, versus 33.2% for the baseline and 37.8% for binary-only DIRECTOR.The combined-feedback approach gives the best results among the tested methods for both base models in automatic and human evaluations.Source: §5 · p. 7, Table 3 · p. 8

Iterative deployment and feedback collection

  • Training modular-plus-binary DIRECTOR on v2 data yields the best results across validation, seen-v1 test, and unseen splits.The v2 data was collected during evaluation of models trained with v1 data.Source: Table 4 · p. 8, §5 · p. 9

Very large models benefit from feedback from smaller models

  • BlenderBot 3-175B, fine-tuned with modular feedback data collected from smaller models, achieves the best human metrics: 64.8% good responses and a 4.08 rating.The authors also report that even this model still makes factual errors and contradictions.Source: Table 3 · p. 8, §5 · p. 9

6 Conclusion

The paper studies whether dialogue models can learn skills from human interaction during deployment and compares feedback types and learning algorithms. It concludes that modular feedback combined with binary feedback in DIRECTOR yields the best results, with further gains from iterative retraining and evidence of usefulness for larger models.

6 Conclusion

  • The paper evaluates whether conversational models can learn new skills through human interaction during deployment, beyond standard pre-training and fine-tuning.Source: §6 · p. 9
  • Granular modular feedback can yield strong performance, and combining it with binary feedback in DIRECTOR yields the paper’s best results.Source: §6 · p. 9
  • The conclusion reports further gains from iterative retraining and redeployment, and says collected feedback is useful for models different from those originally used, including much larger models.Source: §6 · p. 9

7 Limitations and Discussion

The experiments used U.S.-based Mechanical Turk crowdworkers and English responses, a setting the authors distinguish from public deployment with organic users. They also flag feedback-related safety concerns in public use.

7 Limitations and Discussion

  • The evaluation used English responses from U.S.-based Mechanical Turk crowdworkers, unlike public deployments with organically engaged users.Workers were paid above minimum wage, and tasks did not request personal information.Source: §7 · p. 9
  • Detailed feedback could deter organic users, reducing engagement and collected data; the authors suggest natural free-form or lightweight feedback may suit that setting.The study compares feedback methods but could not factor in the tradeoffs of public deployment.Source: §7 · p. 9
  • Noisy or adversarial inputs and feedback raise safety concerns in public deployment, including the risk of misinformation or toxic behavior; steps are addressed in a separate study (Ju et al., 2022).Source: §7 · pp. 9–10

B Success and Failure Cases

Deployment examples contrast BB3-175B’s successful use of search with OPT-175B’s weaker search behavior, and show both feedback-assisted successes and remaining failures.

B Success and Failure Cases

Loading 2208.03270v2…