PaperlayerView exact revision on arXiv

Source-linked AI summary

Deploying Lifelong Open-Domain Dialogue Learning

Kurt Shuster, Jack Urbanek, Emily Dinan, Arthur Szlam, Jason Weston

arXiv:2008.08076v2cs.AIcs.CL

TL;DR

Static training freezes models without language interaction; interactive experiments can be costly, and paid crowdsourcing may yield limited data or a distribution mismatch (de Vries et al., 2020). The authors deploy a fantasy role-playing game where people converse with models, then collect those interactions, retrain models, and redeploy them. Models improve on offline automatic metrics and human continuation rates, while game-collected data costs 1/5th as much per utterance as crowdsourcing and is more effective per utterance at improving continuation rates.

  • Problem

    Static training freezes models without language interaction; interactive experiments can be costly, and paid crowdsourcing may yield limited data or a distribution mismatch (de Vries et al., 2020).

    Source: §1 · p. 1

  • Method

    The authors deploy a fantasy role-playing game where people converse with models, then collect those interactions, retrain models, and redeploy them.

    Source: §1 · p. 1

  • Results

    Models improve on offline automatic metrics and human continuation rates, while game-collected data costs 1/5th as much per utterance as crowdsourcing and is more effective per utterance at improving continuation rates.

    Source: §1 · p. 1

  • Takeaways & Limitations

    The work provides evidence that lifelong dialogue learning with intrinsically motivated humans can succeed, particularly when embedded in games.

    Source: §1 · p. 2

    The dungeon-master model used to assess role-playing is imperfect, making its feedback signal noisy.

    Source: §3 · p. 3

Abstract

from arXiv · show

Much of NLP research has focused on crowdsourced static datasets and the supervised learning paradigm of training once and then evaluating test performance. As argued in de Vries et al. (2020), crowdsourced data has the issues of lack of naturalness and relevance to real-world use cases, while the static dataset paradigm does not allow for a model to learn from its experiences of using language (Silver et al., 2013). In contrast, one might hope for machine learning systems that become more useful as they interact with people. In this work, we build and deploy a role-playing game, whereby human players converse with learning agents situated in an open-domain fantasy world. We show that by training models on the conversations they have with humans in the game the models progressively improve, as measured by automatic metrics and online engagement scores. This learning is shown to be more efficient than crowdsourced data when applied to conversations with real users, as well as being far cheaper to collect.

1 Introduction

The paper addresses the limits of static training and paid crowdsourcing by deploying a fantasy role-playing game where people converse with learning models. Models trained on these interactions improve, while game-collected data is cheaper and more effective per utterance at improving continuation rates.

1 Introduction

  • Static training prevents models from learning through language interactions, while crowdsourcing is costly and may yield data that does not match the desired distribution (de Vries et al., 2020).Crowdsourced data volume is also limited by research budgets.Source: §1 · p. 1
  • The system repeatedly collects human-model conversations, retrains models on new data, and redeploys them, using player continuation rate to compare models online.Continuation rate measures how long players keep playing.Source: §1 · p. 1
  • Game-collected data costs 1/5th as much per utterance as crowdsourcing and is more effective per utterance at improving player continuation rates.The authors attribute the effectiveness to the data being more on-distribution.Source: §1 · p. 1
  • The work provides evidence that lifelong dialogue learning with intrinsically motivated humans can succeed, particularly when embedded in games.The authors report improvements in both offline automatic metrics and human continuation rates.Source: §1 · p. 2

2 Related Work

Prior work spans static open-domain dialogue datasets, lifelong learning, and interactive dialogue learning, but deployed systems accessible for reproducible research remain limited. This paper studies learning from user-facing dialogue in LIGHT, a fantasy role-playing setting.

Open-Domain Dialogue

  • Open-domain dialogue is commonly studied with supervised learning on static train, validation, and test datasets, including crowdsourced or scraped corpora (Huang et al., 2020).Examples include Daily Dialogue, PersonaChat, Empathetic Dialogues, and Wizard of Wikipedia.Source: §2 · p. 2

LIGHT

  • LIGHT situates characters in a medieval fantasy world where they can discuss any topic; its prior crowdsourced dataset contains 8.5k episodes and 111k utterances (Urbanek et al., 2019).The paper uses LIGHT data to build a deployed system for studying lifelong learning from new conversations.Source: §2 · p. 2

Lifelong Learning

  • Lifelong learning lets deployed models interact with the world and improve iteratively, rather than training once on a fixed dataset.The paper distinguishes this from incremental learning, continual reinforcement learning, and never-ending learning.Source: §2 · p. 2

Continual Dialogue Learning

  • Unlike dialogue learning from scalar rewards, this work learns from open-domain interactive dialogue messages (Liu and Lane, 2017; Serban et al., 2017).Source: §2 · p. 2
  • Unlike self-feeding chatbot work using paid crowdworkers, this study evaluates a deployed user-facing system in a rich open-domain environment.Many deployed assistants are inaccessible or unsuitable for reproducible research because of proprietary, commercial, or privacy constraints; the Alexa challenge is a partial exception (Ram et al., 2018).Source: §2 · p. 2

3 Open-domain dialogue as a game

The authors turn open-domain dialogue into a role-playing game in which human players converse with model-controlled characters. Game mechanics encourage role-play and continued play, while scores provide feedback and help motivate participation.

Core Game

  • The game pairs a human and a dialogue agent in a setting, assigning each a character with a name, backstory, and location; one episode is a mini-game.Players role-play their characters in English.Source: §3 · p. 3

Role-Playing (Acting) Score

  • A learned dungeon-master model scores each human utterance by how likely it is given the context, then presents a 1–5-star acting score to reward role-play.The game is designed to train and evaluate open-domain dialogue agents.Source: §3 · pp. 2–3
  • The acting signal is noisy because the dungeon-master model is imperfect, although it provides motivating feedback to players.Source: §3 · p. 3

Other Gamification Steps

  • Players receive role-playing scores, leaderboard rankings, and character badges as additional incentives to play well and try different characters.Source: §3 · p. 3

Game Loop

  • Each mini-game has six turns per agent, after which players can choose another location, wait for a character, switch characters and setting, or end the game.The varied choices are intended to support diverse dialogue data.Source: §3 · pp. 3–4

Game Safety

  • A safety classifier checks human and model turns, and retrieval models let the team vet all response candidates before runtime; game settings and personas are also vetted.Players agree to public data release, are told not to share personally identifying information, and only their game messages are retained.Source: §3 · p. 4

4 Lifelong Dialogue Learning

The learning setup combines retrieval and generative models with training data drawn from human turns in deployed conversations. It iterates collection and training, and uses randomized model deployment to compare engagement online.

4.1 Models

  • The deployed retrieval models use Poly-Encoder Transformers, while offline generative evaluation uses a fine-tuned 2.7-billion-parameter BlenderBot model.Retrieval models include 90M- and 622M-parameter variants pre-trained on 1.5B Reddit examples; negative-context training and decoding control are also considered.Source: §4.1 · p. 4

4.2 Iterative Data Collection and Training

  • Training data consists of context and next-utterance pairs with human responses, treated as gold because engaged humans are assumed to provide high-quality turns despite model mistakes.The authors also test filtering out episodes whose summed acting score falls below a validation-tuned threshold.Source: §4.2 · p. 5
  • The collection-training cycle redeploys each newly trained model, under the hypothesis that better models yield higher-quality human data and longer conversations.Source: §4.2 · p. 5

4.3 Deployment-based Evaluation

  • For online evaluation, a random model from a pool is assigned per mini-game, and models are compared by the average rate at which players continue to another mini-game.This continue rate also supports online model selection.Source: §4.3 · p. 5

5 Experiments

The experiments implement lifelong learning in three rounds, adding conversations collected from earlier deployments to later models’ training data.

5.1 Rounds of Learning

  • Round 1 models were trained only on LIGHT MTurk data and deployed in the game.The deployed models were retrieval-model variants.Source: §5.1 · p. 5
  • Round 2 models added 50,982 examples collected from the Round 1 deployment to LIGHT MTurk data before redeployment.Source: §5.1 · p. 5
  • Round 3 models used LIGHT MTurk data, the 50,982 Round 1 examples, and 180,010 additional examples from Round 2.Source: §5.1 · p. 5

5.2 Data Collection

LIGHT WILD is a diverse collection of human-model dialogue gathered through deployment; the authors froze collection for the reported experiments and data release.

5.2 Data Collection

  • Collection was frozen for the paper’s experiments and data release, with total newly collected LIGHT WILD statistics reported in Table 1.Source: §5.2 · p. 5, Table 1 · p. 6
  • The collection involved over 13,000 players and includes 587 unique locations, 630 roles, and more unique tokens than the compared dataset.Source: §5.2 · pp. 5–6
  • Validation and test sets used complete Round 2 conversations in which players scored at least 9 stars.The authors selected these conversations to build higher-quality evaluation sets.Source: §5.2 · p. 5
  • LIGHT WILD contains around twice the human utterances and 4x the dialogue utterances of compared datasets, at 1/5 the LIGHT MTurk cost per utterance.Source: Table 2 · p. 6

5.3 Analysis of Results

Across successive rounds, models trained on collected dialogue improve on offline metrics, while the learning curves, deployment evaluations, and data analyses characterize the gains in cost and user engagement. WILD data also differs in quality and content from crowdsourced dialogue.

5.3.1 Performance by Round

  • 87.72% (Seen) and 83.48% (Unseen), versus LIGHT’s 76.5% and 70.5%, respectively; across three rounds, both model types improve on both metrics across all test sets.The rounds show steady Hits@1/20 gains for retrieval models and PPL decreases for generative models, including gains on LIGHT (MTurk) tests despite WILD’s different distribution.Source: §5.3.1 · p. 6, Table 3 · p. 7

5.3.2 Lifelong Learning Curves

  • At LIGHT’s training-set size, WILD reaches 83.59% vs. LIGHT’s 80.63%; all WILD data reaches 85.95%, and mixing sources reaches 87.2%.WILD’s per-example gains have not saturated, and using all data from both sources yields the best performance.Source: §5.3.2 · p. 6, Figure 2 · p. 7

5.3.3 Cost Learning Curves

  • At equal spend, WILD reaches 85.95% vs. LIGHT MTurk’s ∼77.5%; matching 80.63% costs over 8× more with MTurk.WILD costs around one-fifth as much per utterance as LIGHT MTurk.Source: §5.3.3 · p. 7, Figure 2 · p. 7

5.3.5 Data Quality

  • Higher predicted-quality bins monotonically improve WILD validation accuracy; filtering low bins hurts, reducing 87.06% to 86.69% when removing bins below 6 and to 85.38% when removing bins below 9.The quality-bin comparison adds 10,000 WILD examples from each bin to the 111k LIGHT MTurk examples.Source: Figure 3 · p. 8, §5.3.5 · p. 9

5.3.4 Deployment-based Evaluation

  • Across model sizes, LIGHT+WILD, negative-context training, and decoding control improve continuation; larger models nevertheless fare worse, often seeming boring despite fewer mistakes.Continue rates across the tested models range from 68% to 75%; the combined enhancements improve more than either alone.Source: §5.3.4 · pp. 7–8, Table 4 · p. 8, Table 5 · p. 8

5.3.6 Observations on Gamification

  • Adding three post-episode controls raised continuation by 3.3 ± 1.6%; better-liked models also lower collection costs by increasing player engagement.The reported increase compares the three controls with a single choice using the same model; model comparisons are made within the same batch.Source: §5.3.6 · p. 9

5.3.7 Analysis of Data Distribution

  • WILD players favor exciting, emotional, action-packed exchanges, while crowdworkers more often discuss dry topics such as harvests and taxes.Source: §5.3.7 · p. 9

6 Conclusion and Future Work

The paper presents a deployed game for lifelong dialogue learning, where training on human interactions improves offline metrics and user engagement. It proposes future work on learning signals, difficult-case collection, and extending role-play to other domains.

6 Conclusion and Future Work

  • The deployed game supports lifelong open-domain dialogue learning, and training on its human conversations improves offline metrics and user engagement.The authors describe this as evidence that models can continually improve by learning from people in the wild rather than only from paid crowdworkers.Source: §6 · p. 10
  • Future work could explore other learning methods or signals in the released data, including signals extracted from human utterances.The authors also suggest using game-engine control to emphasize difficult or informative cases if model performance begins to saturate.Source: §6 · p. 10
  • The role-playing setup could be extended to other domains and include both dialogue and actions.Source: §6 · p. 10

A Game Screenshots

Figures 4 and 5 show screenshots of the game.

A Game Screenshots

B Using WILD Model responses

The experiments compare training on human versus model utterances from WILD. Human utterances achieve higher Hits@1/20 performance on both WILD and LIGHT validation sets.

B Using WILD Model responses

  • The main results use human utterances from the role-playing game as WILD training targets because model utterances may contain mistakes.The authors also consider training on model utterances as a possible distillation of the previous model's knowledge.Source: B Using WILD Model responses · p. 10
  • 86.05% for human utterances versus 73.96% for model utterances on WILD, and 82.32% versus 77.56% on LIGHT, in Hits@1/20.Source: B Using WILD Model responses · p. 10

C Comparing Small and Large Model Variants

Crowdworker evaluations found that the larger retrieval model avoided several common mistakes and had more conversations without listed issues, but was also rated repetitive or boring more often. The authors identify a trade-off between fewer mistakes and engagement.

C Comparing Small and Large Model Variants

  • Crowdworkers held 12-turn conversations with each model, evaluated utterances for common mistakes, and rated how enjoyable role-playing was.The evaluated models were LW90M and LW622M, trained with LIGHT + WILD data without decoding control.Source: C Comparing Small and Large Model… · p. 10
  • 34.1% of LW622M conversations had no listed issues, compared with 27.3% for LW90M.LW622M also contradicted itself less, used the wrong character or location less often, and went off-topic less frequently.Source: C Comparing Small and Large Model… · p. 10, Table 6 · p. 11
  • LW622M was rated repetitive or boring more than twice as often as LW90M, and its utterances averaged two words shorter.The authors suggest this may help explain why players were more likely to leave after interacting with the larger model.Source: C Comparing Small and Large Model… · p. 10
  • The authors identify a scaling trade-off: fewer mistakes did not prevent the larger model from becoming more boring in live deployment.Source: C Comparing Small and Large Model… · p. 11
Loading 2008.08076v2…