PaperlayerView exact revision on arXiv

Source-linked AI summary

Human-centric Dialog Training via Offline Reinforcement Learning

Natasha Jaques, Judy Hanwen Shen, Asma Ghandeharioun, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Shane Gu, Rosalind Picard

arXiv:2010.05848v1cs.CLcs.LG

TL;DR

Offline dialog learning must handle sparse coverage and uncertainty while improving conversations from human feedback. The paper uses KL-control and pessimistic value estimates in an offline RL method, which optimizes rewards and elicits human responses. The authors report higher overall human ratings and positive feedback than existing approaches.

  • Problem

    Offline RL cannot explore beyond its fixed batch, and missing state-action pairs can cause arbitrarily bad value estimates.

    Source: §3.1 · p. 3

  • Method

    WOP constrains offline RL updates with KL-control from a pretrained language model and uses pessimistic estimates under uncertainty.

    Source: §1 · p. 2

  • Results

    The method successfully optimizes generated bot rewards and elicited human responses, and presents a better option than regularization for training a specific bot behavior.

    Source: §7 · p. 8

  • Takeaways & Limitations

    The techniques enable offline RL on existing base language models from real human conversations, supporting post-hoc fine-tuning for desired behaviors.

    Source: §7 · p. 8

    Which RL objective improves overall qualitative ratings remains open, and manual ratings are too sparse to optimize effectively.

    Source: §7 · p. 8

Abstract

from arXiv · show

How can we train a dialog model to produce better conversations by learning from human feedback, without the risk of humans teaching it harmful chat behaviors? We start by hosting models online, and gather human feedback from real-time, open-ended conversations, which we then use to train and improve the models using offline reinforcement learning (RL). We identify implicit conversational cues including language similarity, elicitation of laughter, sentiment, and more, which indicate positive human feedback, and embed these in multiple reward functions. A well-known challenge is that learning an RL policy in an offline setting usually fails due to the lack of ability to explore and the tendency to make over-optimistic estimates of future reward. These problems become even harder when using RL for language models, which can easily have a 20,000 action vocabulary and many possible reward functions. We solve the challenge by developing a novel class of offline RL algorithms. These algorithms use KL-control to penalize divergence from a pre-trained prior language model, and use a new strategy to make the algorithm pessimistic, instead of optimistic, in the face of uncertainty. We test the resulting dialog model with ratings from 80 users in an open-domain setting and find it achieves significant improvements over existing deep offline RL approaches. The novel offline RL method is viable for improving any existing generative dialog model using a static dataset of human feedback.

1 Introduction

Open-domain dialog models lack a single correct response, and learning from live human interactions must be done offline to test policies before deployment. The paper introduces WOP and implicit conversation rewards, reporting improved ratings against offline RL baselines.

1 Introduction

  • Training from live human interactions offline allows policies to be tested before deployment, reducing the risk of learning inappropriate behaviors.The authors host dialog models for real-time user conversations and optimize implicit feedback with offline reinforcement learning.Source: §1 · p. 1
  • Offline RL risks extrapolation error when the fixed dataset does not cover state-action pairs, producing arbitrarily bad value estimates.The paper positions WOP as a response to the difficulty of exploration and over-optimistic estimates in offline learning.Source: §1 · p. 1
  • The method learns from multiple reward functions in a 20,000-token action space.The paper develops rewards to approximate implicit human preferences expressed in conversational responses.Source: §1 · p. 2
  • 80 new human conversation partners rated the model more positively and gave higher quality ratings than a state-of-the-art offline deep RL method.The method optimizes reward functions designed to approximate implicit human preferences expressed during conversation.Source: §1 · p. 2
  • WOP combines KL-control from a pretrained language model with pessimistic estimates under uncertainty to address offline RL challenges.KL penalties keep the learned policy close to realistic language, while dropout estimates uncertainty in target Q-values to reduce over-optimistic bias.Source: §1 · pp. 1–2

2 Related Work

Prior open-domain dialog work explored restricted RL actions, explicit feedback, and implicit signals, but offline RL on real human interactions remained relatively unexplored. This work relates KL-control to offline learning without exploration and integrates a pretrained model into the policy.

2.1 Dialog

  • Open-domain dialog RL was less explored than task-oriented RL, while prior work used restricted actions, explicit feedback, or implicit signals in MLE and on-policy settings.The authors state that, to their knowledge, theirs is the first use of offline RL to train dialog models on real human interactions.Source: §2.1 · p. 2

2.2 Offline RL and KL-Control

  • Unlike methods that rely on exploration or policy evaluation, this work learns entirely from a fixed batch; KL-control from a pretrained model directly incorporates prior knowledge into the RL policy.BCQ constrains actions near the offline data using a learned generative model, whereas this work integrates the prior model through KL-control.Source: §2.2 · pp. 2–3

3 Way Off-Policy RL

The section frames conversation generation as offline RL, where token-level actions are learned from human-response rewards using a fixed interaction dataset. WOP combines a language prior with uncertainty-aware pessimism to constrain learning.

3 Way Off-Policy RL

  • For dialog, conversation history is the state, each generated token is an action, and the human response supplies a reward over the conversation trajectory.The policy is trained to maximize discounted reward.Source: §3 · p. 3

3.1 Offline RL and extrapolation error

  • Offline RL learns from a fixed batch without further environment interaction, but unsupported state-action pairs can produce arbitrarily bad value estimates.Bellman bootstrapping can accumulate those errors; offline learning also lacks the exploration that normally helps refine uncertain estimates.Source: §3.1 · p. 3

3.2 Dropout for uncertainty estimation of Target Q-values

  • Dropout-based stochastic target-Q estimates use their minimum to penalize high-variance estimates and favor states and actions covered by offline data.This makes the algorithm pessimistic rather than optimistic under uncertainty.Source: §3.2 · p. 4

3.3 KL Control from pre-trained prior

  • KL-control regularizes the learned policy against a prior language model, favoring realistic actions likely to occur in the offline data while retaining entropy-based diversity.The section relates this objective to soft Q-learning, or Ψ-learning, whose updates avoid a hard maximum over noisy estimates.Source: §3.3 · pp. 4–5

3.4 Comparison to existing techniques

  • The section compares WOP with discrete BCQ, using a trained language model as BCQ’s action prior and Monte Carlo target estimation to reduce overestimation.The comparison also removes BCQ’s continuous-action perturbation model for the discrete action space.Source: §3.4 · p. 5

4 Learning from talking to humans

The study collects open-ended human-model conversations through a real-time platform, then evaluates trained bots with ratings from new users. The setup includes baseline models from two conversational datasets and filtering of online interactions before offline RL training.

4.1 Training baseline dialog models

  • Baseline dialog models were trained on CORNELL movie dialog and REDDIT Casual Conversations using hierarchical sequence-to-sequence architectures.The authors report training more than 40 models across architectures, datasets, and feature-based regularization settings; the proposed techniques are model-agnostic.Source: §4.1 · p. 5

4.2 Hosting real-time conversations online

  • The online platform hosted dialog models for real-time conversation, with response upvotes and downvotes and post-chat Likert ratings.The ratings covered conversation quality, fluency, diversity, contingency/relatedness, and empathy.Source: §4.2 · p. 5
  • From 46,061 user-input and agent-response pairs, the authors filtered out one-character responses, profanities, and invalid inputs, leaving 45,179 pairs for offline RL.The filtering was intended to prevent the RL algorithms from learning undesirable human behavior.Source: §4.2 · p. 5

4.3 Evaluating offline RL models

  • The authors recruited 80 Mechanical Turk workers to rate bots after interacting with each for at least six turns, collecting 600 seven-point Likert ratings.Ratings covered conversation quality, fluency, diversity, contingency/relatedness, and empathy.Source: §4.3 · p. 5, §4.2 · p. 5

5 Measuring implicit conversation cues

The section develops reward functions from implicit conversational cues because identifying human satisfaction is unresolved and manual ratings may be cumbersome. It compares reward scores with users’ manual votes and optimizes only cues that map accurately to those ratings.

5 Measuring implicit conversation cues

  • The authors motivate implicit cues as a less cumbersome alternative to requiring humans to manually rate good interactions, while noting that suitable training goals remain an open question.Their stated goal is to improve natural conversation by learning from signals in human responses.Source: §5 · p. 6
  • The authors define 30 bot-, user-, and interaction-based rewards, covering model behaviors, elicited user behaviors, and user-response similarity.Examples include asking questions, eliciting laughter, and similarity in sentence-embedding space.Source: §5 · p. 6
  • Only some candidate rewards mapped accurately to manual user ratings, and those were selected for RL optimization.Conversation length and specificity were not higher in upvoted bot responses.Source: §5 · p. 6

6 Results

KL-control models outperform offline RL baselines in overall human ratings and elicit positive human responses, but RL models show more repetition. Among individual rewards, user sentiment and repetition minimization support high-quality conversations.

6.1 Controlling bot conversation behavior

  • KL-control achieves higher bot sentiment than VHRED and VHRED-EI, and controlling bot sentiment also elicits higher user sentiment.Figure 3a compares average sentiment reward scores on study chat transcripts.Source: §6.1 · p. 7, Figure 3 · p. 7

6.2 Measuring human conversation behavior

  • KL-control models elicit higher user sentiment and laughter rewards than other offline RL algorithms and the VHRED baseline.Figure 3b reports sentiment and laughter elicited from human participants.Source: §6.2 · p. 7, Figure 3 · p. 7

6.4 The role of repetition

  • Offline RL bots score lower in overall human quality than the language-model prior and show more conversation and utterance repetition.The authors hypothesize limited offline data may narrow outputs toward high-reward conversations, increasing repetition.Source: §6.4 · p. 7, Figure 3 · p. 7

6.3 Overall human ratings

  • KL-control models significantly outperform Batch Q and DBCQ on total human ratings, with substantial gains over RL baselines across ratings and human reward.The reported one-way ANOVA gives F(x) = 7.328, p < .005.Source: §6.3 · p. 7, Table 2 · p. 8

6.5 Comparing rewards

  • Maximizing positive user sentiment yields the highest-quality bot among the reward comparisons; manual-vote training earns decent quality but no higher upvote z-score than other models.The manual-vote reward may be difficult to optimize directly because users did not vote often enough to provide a good signal.Source: §6.5 · p. 8, Table 3 · p. 8
  • Minimizing repetition at both conversation and utterance levels achieves the best overall quality among bots trained on single features.Implicit enjoyment signals such as sentiment are dense, making them a more scalable way to learn from preferences than sparse manual votes.Source: §6.5 · p. 8

7 Discussion

The paper presents offline RL techniques for shaping dialog behavior from real human conversations, and reports improved rewards and human responses compared with prior approaches. The appropriate reward objectives for improving overall qualitative ratings remain unresolved, especially given the limited offline data and the narrow coverage of the proposed rewards.

7 Discussion

  • The method successfully optimizes generated bot rewards and elicited human responses, and is reported as a better option than training a specific behavior through regularization.The paper says RL remains the only option for maximizing user feedback over a conversation.Source: §7 · p. 8
  • WOP achieves higher performance than prior offline RL methods on traditional tasks, elicits more positive feedback from novel humans, and earns higher overall human ratings.The paper also reports that its reward set did not achieve higher human quality ratings with the limited offline training data collected.Source: §7 · pp. 8–9
  • The study leaves open which RL objectives improve overall qualitative ratings; its proposed reward set was insufficient with the limited offline data collected.The authors note that the rewards may not cover all aspects of high-quality open-ended conversation and suggest investigating long-term rewards.Source: §7 · pp. 8–9
  • The rewards use dialog data and annotations from online task workers in the United States; broader deployment calls for representative, diverse conversations and annotations.Source: §7 · p. 9
  • The techniques shape dialog behavior toward a desired objective, such as appropriate, positive, or polite language, even if some users perceive lower conversation quality.Source: §7 · p. 9
  • WOP teaches specific behaviors from offline data more effectively than previously proposed RL or regularization techniques.Source: §7 · p. 9

A.1 Training details and hyperparameters

The experiments use VHRED-based dialog models trained on movie-dialog and Reddit data, then apply offline RL initialized from the Reddit-trained model. RL training uses tuned shared reward weights and substantially less training time than the VHRED baseline.

A.1 Training details and hyperparameters

  • Baseline models use VHRED, with an enhanced version adding sentiment and semantic-content prediction layers through knowledge distillation.The models were trained on movie dialogs and Reddit casual conversations; the enhanced VHRED has 95.4 million parameters versus 76.6 million for VHRED.Source: Baseline Models · p. 12

RL Models

  • RL models were initialized from the best Reddit-trained model and trained for 2,000 epochs, with early stopping and manually selected checkpoints.The best KL-Control Q and KL-Control Ψ checkpoints were selected at 1,600 and 1,800 epochs, respectively; reward weights were manually tuned, with only three sets tested.Source: RL Models · pp. 12–13

A.2 Computing Infrastructure

  • Training an RL model for 2,000 epochs took about 30 minutes, compared with around six hours for a VHRED baseline.Each RL model was trained on a NVIDIA GeForce GTX 1080 GPU.Source: §A.2 · p. 13

A.3 Model Validation and Evaluation

Models are evaluated through interactive chats in which participants rate conversation qualities, and the authors also use chat ratings while tuning models. In VHRED-EI experiments, KL-control performs better than other offline RL algorithms but does not exceed the VHRED-EI baseline in qualitative ratings.

A.3 Model Validation and Evaluation

  • Participants chatted with each bot for at least six human responses, then rated quality, fluency, diversity, contingency, and empathy on a 7-point Likert scale.Amazon Mechanical Turk participants rated seven or eight bots each; the authors also interacted with and rated bots during validation.Source: §A.3 · p. 13

B Offline-RL with VHRED with Emotion and Infersent Regularization

  • On VHRED-EI, KL-control models received higher qualitative ratings than other offline RL algorithms, but no RL model surpassed VHRED-EI.When trained with individual rewards, KL-Control Ψ elicited the highest qualitative ratings with User Sentiment.Source: B Offline-RL with VHRED with Emotion… · p. 13

C Traditional RL experiments

The traditional RL experiments compare offline algorithms across replay-buffer conditions using CartPole and Acrobot. WOP outperforms the compared algorithms in CartPole, while the supplied passages do not state an Acrobot outcome.

C Traditional RL experiments

  • A shared VAE models the action prior p(a|s) for DBCQ and WOP and can also generate actions for behavioral cloning.The experiments benchmark these approaches against batch Q-learning; CartPole uses the Ψ-learning loss and Acrobot the traditional Q-learning loss.Source: C Traditional RL experiments · pp. 13–14
  • The experiments run 50 random-seed trials per model; the full-buffer condition supplies 20,000 experience samples.Source: C Traditional RL experiments · p. 14
  • The four buffer conditions vary data coverage and behavior quality, from all online experience to expert-only or noisy-demonstrator samples.The concurrent condition uses a sliding window of experience in the online learner’s order; the noisy demonstrator acts randomly with probability 0.3.Source: C Traditional RL experiments · p. 14
  • Across CartPole offline conditions, WOP outperforms Batch Q, behavioral cloning, DBCQ, and the behavior policy.The comparison covers full-buffer, concurrent, expert-demonstrator, and noisy-demonstrator data conditions.Source: C Traditional RL experiments · pp. 14–15, Figure 4 · p. 15

D Additional results

The additional results link KL-control to lower divergence from the prior during training and identify implicit conversational signals that correlate with manual votes. The appendices describe how these rewards are computed and how the interactive platform and server were configured.

D Additional results

  • With KL-control, policy divergence from the prior stays lower throughout training; without KL regularization, baseline models diverge and DBCQ selects unrealistic actions.The authors connect DBCQ's divergence to unrealistic Q-estimates and generated utterances.Source: Figure 6 · p. 15, D Additional results · p. 15

E Implicit Rewards Details

  • User Laughter and User Sentiment, Bot Sentiment, Bot Laughter, Bot Convo. Repetition, Bot Utterance Repetition, Word Similarity, and USE Similarity correlate with manual votes.The figures show mean z-scores for user-response, bot, and bot-user rewards by manual vote.Source: E Implicit Rewards Details · p. 15, Figure 8 · p. 17, Figure 9 · p. 18

E.1 Sentiment-based

  • Sentiment rewards use weighted emoji emotion and sentiment trajectories; engagement rewards discount conversation length across turns, while laughter rewards count laughter strings in user responses.Bots trained to maximize user laughter became extremely supportive and cheerful.Source: §E.1 · p. 16, §E.2 · p. 16, §E.3 · p. 16

E.4 Semantic similarity

  • Semantic-similarity rewards use sentence embeddings or word overlap, while Bot Question rewards question words and question marks in bot utterances.The similarity rewards are intended to encourage on-topic responses; Bot Question awards 0.5 for a question word and another 0.5 for a question mark.Source: §E.4 · p. 16, §E.5 · p. 16

E.6 Phrase based rewards

  • Post-hoc phrase counts measure polite, cheerful, and supportive language, while a toxicity classifier supplies the Bot Toxicity reward.The phrase metrics were designed after the authors observed shifts toward polite, cheerful, and supportive speech.Source: §E.6 · p. 16, §E.7 · p. 17

E.8 Specificity

  • Without a knowledge-graph back end, specificity is limited; normalized IDF rewards specificity, while negated non-unique-word counts penalize utterance and conversation repetition.Source: §E.8 · p. 18, §E.9 · p. 18

F Interactive bot platform details

  • The GPU-hosted platform allowed real-time chats and ratings after at least three turns; its current chat corpus contains personally identifying information and cannot be released in its current form.Source: F Interactive bot platform details · p. 18

F.1 Website server setup and configuration

  • Keeping bots in the Django process required single-process uWSGI and substantial memory; the 64GB-RAM, GPU setup supported hundreds of simultaneous users and more than 30 bots.Source: §F.1 · pp. 18–19
Loading 2010.05848v1…