Source-linked AI summary
Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog
Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, Rosalind Picard
TL;DR
The paper addresses the difficulty of learning reinforcement-learning policies from fixed, heterogeneous human-interaction data without online exploration. It combines pre-trained priors, KL-control, and dropout-based uncertainty estimates for offline batch RL, and reports improved performance in open-domain dialog, including significant gains from KL-control over baseline methods. The approach also supports post-hoc learning from multiple implicit human-reward functions.
Problem
Deep RL often struggles with off-policy data without exploration, although offline learning is important when interaction is expensive and policies must be tested before deployment.
Method
The paper uses pre-trained generative models as priors, KL-control to regularize policy divergence, and dropout-based lower bounds for target Q-values in batch RL.
Results
KL-control models show substantial gains over baseline models across ratings and human reward, with significantly better total human rating scores than Batch Q baselines and DBCQ (F(x) = 4.781, p < .05).
Takeaways & Limitations
Way Off-Policy can learn effectively from heterogeneous human-interaction batches and support multiple reward functions designed post-hoc to extract implicit preferences.
Takeaways & Limitations
The proposed rewards improve elicited human reward but not quality ratings relative to the prior because they do not fully cover high-quality conversation.
Abstract
from arXiv · showhide
Most deep reinforcement learning (RL) systems are not able to learn effectively from off-policy data, especially if they cannot explore online in the environment. These are critical shortcomings for applying RL to real-world problems where collecting data is expensive, and models must be tested offline before being deployed to interact with the environment -- e.g. systems that learn from human interaction. Thus, we develop a novel class of off-policy batch RL algorithms, which are able to effectively learn offline, without exploring, from a fixed batch of human interaction data. We leverage models pre-trained on data as a strong prior, and use KL-control to penalize divergence from this prior during RL training. We also use dropout-based uncertainty estimates to lower bound the target Q-values as a more efficient alternative to Double Q-Learning. The algorithms are tested on the problem of open-domain dialog generation -- a challenging reinforcement learning problem with a 20,000-dimensional action space. Using our Way Off-Policy algorithm, we can extract multiple different reward functions post-hoc from collected human interaction data, and learn effectively from all of these. We test the real-world generalization of these systems by deploying them live to converse with humans in an open-domain setting, and demonstrate that our algorithm achieves significant improvements over prior methods in off-policy batch RL.
1 Introduction
The paper targets batch RL that can learn offline from heterogeneous historical interaction data, while allowing policies to be tested before real-world deployment. It applies this challenge to open-domain dialog, using implicit human reactions and a strong prior to learn from collected conversations.
- 1 Introduction: Offline batch RL must learn from static, potentially old data without exploring, because real-world interaction is costly and policies may require safety testing.The setting is especially important for systems learning from human interaction.
- 1 Introduction: The proposed approach uses a pre-trained generative state-action model as a prior, KL-control to limit divergence, and dropout-based lower bounds to reduce Q-value overestimation.The dropout method is presented as an alternative to Clipped Double Q-Learning.
- 1 Introduction: The study learns from implicit human reactions rather than explicit feedback, which does not scale to large-scale interaction.The application seeks to infer preferences from human responses during conversation.
- 1 Introduction: Open-domain dialog makes batch RL difficult because language generation has a 20,000-dimensional action space and combinatorial state-space growth.These properties compound overestimation and make offline learning harder.
- 1 Introduction: A batch collected from over 40 dialog models supports learning with Way Off-Policy despite heterogeneous architectures and training datasets.The same data can support multiple reward functions designed post-hoc to extract implicit human preferences.
2 Related Work
Related work situates the paper at the intersection of KL-regularized control, off-policy learning, uncertainty estimation, and reinforcement learning for dialog. The paper distinguishes its setting by focusing on batch learning without exploration and implicit feedback in open-domain conversation.
- KL-control: KL-control regularizes an RL policy by penalizing its KL divergence from a reference distribution, with prior uses including conservative policy updates and entropy maximization.The paper positions its use of a strong pre-trained prior as novel for batch RL without exploration.
- Off-policy learning: Many off-policy methods address distribution shift or value divergence in settings where policies can explore and collect additional data.The paper instead focuses on fixed-batch learning without exploration.
- Reward learning: The paper notes that effective off-policy batch learning could enable learning from multiple rewards post-hoc, potentially supporting reward relabeling techniques.This possibility depends on learning successfully from the same collected batch.
- Uncertainty estimation: Dropout-based uncertainty estimation belongs to a broader line of RL methods using uncertainty estimates as alternatives or complements to Double DQN.The paper applies dropout specifically to target Q-values in batch RL.
- Dialog reinforcement learning: Prior dialog RL research largely targets task-oriented systems with limited action sets, while open-domain applications are less numerous.Human input in earlier work is often explicit, though some studies use implicit signals such as interruptions or restarting.
3 Methods
The methods combine pre-training, batch Q-learning, uncertainty-aware targets, discrete batch constraints, KL-control, entropy regularization, and model averaging. Together, these components keep learning near data-supported actions while reducing optimistic value estimates and incorporating prior behavior.
- Batch Q-learning: Batch Q-learning pre-trains a generative p(a|s) model and initializes the Q-networks from it before updating on sampled batch transitions.This baseline uses the pre-trained model to reduce variance and combat overestimation bias.
- Batch overestimation: In batch RL, noisy estimates for poorly covered state-action pairs make the max operator overestimate values and favor regions unsupported by the data.The absence of exploration prevents the policy from refining those estimates.
- Dropout uncertainty estimation: Dropout produces a distribution of target-Q predictions, whose lower bound is estimated by M stochastic forward passes with independent dropout masks.Taking the minimum penalizes high-variance estimates and induces pessimism toward uncertainty.
- Discrete Batch Constrained Q: DBCQ adapts Batch Constrained Q-learning to discrete actions by using a strong pre-trained prior instead of a continuous-action perturbation model.The prior is motivated as more robust than one learned from a small human-interaction batch.
- KL Control from pre-trained prior: KL-control directly incorporates the prior p(a|s) into the Q-learning policy by penalizing divergence while continuing to maximize reward.The prior favors realistic actions likely to occur in the batch, while −log π(a|s) supplies entropy regularization.
- Ψ-learning: Ψ-learning replaces hard maximization with a soft Boltzmann-based value function, reducing overestimation and stabilizing temporal-difference updates.The method is proposed as especially useful for reducing optimism under batch uncertainty.
- Model averaging: Model averaging constructs a prior from weighted predictions of multiple models with different architectures and data histories.The weights can reflect model quality or the proportion of batch data generated by each model.
4 RL for open-domain dialog generation
This section frames open-domain dialog as a difficult batch-RL problem and describes a hierarchical token-generation setup trained from implicit human reactions. Pre-trained language-model structure supplies a prior while human responses provide reward information.
- Dialog model: Hierarchical seq2seq models use encoder, context, and decoder recurrent networks to generate responses token by token.The context network updates after each utterance, while the decoder produces the next utterance one token at a time.
- RL formulation: 20,000 vocabulary actions make dialog generation highly high-dimensional and worsen the difficulty of batch reinforcement learning.The model constructs each response by iteratively selecting the next token.
- RL formulation: The conversation state contains all preceding bot and human text, organized hierarchically into utterances and tokens.Human interaction is treated as the environment, and human responses contribute to reward computation.
- RL formulation: Target future rewards use Q-values for subsequent bot tokens and incorporate the human response after the bot finishes an utterance.The human response is appended to the conversation before estimating the next state.
- Implicit human preferences: Intrinsic rewards target positive sentiment, engagement, laughter, and semantic similarity as implicit signals of conversational quality.These rewards are designed to learn from how humans respond rather than requiring manual labels.
5 Experiments
The experiments collect interactive conversations through a GPU-hosted web platform running diverse neural dialog models trained on different architectures and datasets.
- Data collection: The study trained over 40 dialog models with different architectures on movie-dialog and Reddit datasets.The models varied significantly in their data distributions.
- Data collection: A CUDA-capable web app hosted the dialog models on GPUs for fast, real-time inference during human interaction.The platform was used to collect interactive conversation data.
- Data collection: The collected models differed substantially in architecture and training data, creating varied interaction data for batch learning.
6 Results
KL-control substantially outperformed Batch Q and DBCQ in interactive human evaluation while remaining closer to the pretrained prior. It also produced more plausible, supportive dialogue and enabled post-hoc comparison of multiple implicit reward functions, though the proposed rewards did not fully capture conversation quality.
- Interactive evaluation: KL-control models significantly outperformed Batch Q baselines and DBCQ on total human rating scores.The reported ANOVA found F(x) = 4.781, p < .05.
- Policy behavior: KL-control kept the policy’s KL-divergence from the prior lower throughout training.Figure 2 reports standard-deviation bands around the trajectories.
- Policy behavior: Batch Q exploited simple rewards by repeatedly asking questions and producing maximum-length, implausible utterances.The model generated 30 tokens each utterance and phrases such as “where did you say to me?”.
- Limitations: The proposed rewards improved elicited human reward but did not fully cover high-quality conversation as measured by quality ratings.The authors identify this mismatch as a limitation and motivate learning from improved rewards.
- Policy behavior: KL-control shifted dialogue toward realistic, polite, supportive, and cheerful language while eliciting higher human reward.Figure 3 distinguishes training rewards from post-hoc metrics and reports the human reward as the blue metric.
- Deployment evaluation: KL-control models elicited significantly higher reward than baselines, while KL-control Q performed best overall and for words elicited.The result came from trajectories among the ten best conversations; laughter remained extremely rare.
- Reward functions: Maximizing positive and minimizing negative user sentiment produced the highest-quality bot among the independently trained reward functions.Utterance-level manual votes performed worse because users voted infrequently, supporting implicit sentiment as a more scalable signal.
7 Conclusion
The paper presents techniques for batch RL without exploration, emphasizing KL-control from a strong prior and implicit human rewards for open-domain dialog.
- KL-control from a strong pre-trained prior is proposed to improve batch RL without exploration.
- Implicit human rewards provide rich cues for open-domain dialog learning, with sentiment-based rewards showing the most promise.
- Maximizing implicit rewards outperforms relying on explicit feedback, although the proposed rewards remain incomplete.
8.1 Details about implicit metrics
The study constructs implicit dialog rewards from sentiment, engagement, laughter, semantic coherence, questions, and weighted combinations of these signals.
- Sentiment-based: Sentiment reward weights emoji predictions to produce higher values for positive and lower values for negative sentiment.A sentiment-transition reward additionally rewards conversations whose peak positive sentiment follows peak negative sentiment.
- Engagement-based: Conversation length and user response words serve as engagement-based rewards distributed across utterances.The discounted reward for utterance n < N is γN−nN.
- Laughter-based: Laughter reward counts occurrences of the string ‘ha’ in the user’s response.Bots trained to maximize laughter learned extremely supportive and cheerful language.
- Semantic coherence: Semantic coherence rewards similarity between the user’s input and the bot’s response using sentence embeddings.
- Questions: Question reward assigns 0.5 for a question word and an additional 0.5 for a question mark.
- Combined reward: The total reward combines question, semantic coherence, laughter, sentiment transition, sentiment, words elicited, and conversation length rewards.The component weights range from 0.1199262 to 0.15682657.
- Post-hoc metrics: Post-hoc metrics measure politeness, supportiveness, and cheerfulness by counting whether selected phrases occur in an utterance.The phrase lists include examples such as ‘please,’ ‘you’re right,’ and ‘happy.’
8.2 Training details and hyperparameters
Training used shared RL hyperparameters and a hierarchical VHRED dialog model with bounded conversation and sentence lengths.
- RL models trained for 800–1000 batches with batch size 32 and early stopping selecting the best checkpoint.Shared settings included γ = 0.5, c = 2, M = 5, α = .005, and learning rate r = .0001.
- The VHRED model used context, decoder, and encoder hidden sizes of 1000, 1250, and 1250, respectively.Its z embedding size was 600 and dropout was d = 0.2.
- Conversation context was limited to five utterances and each sentence to 30 tokens.
- The Context RNN included two 128-unit feedforward semantic-prediction layers trained using knowledge distillation.The layers used ReLU activation.
8.3 Additional results
Additional results show that some reward-trained bots generalize to new users, while others do not reliably elicit their intended responses in live interaction.
- Question- and positive-sentiment-trained bots effectively generalize to new users, whereas the words-elicited bot does not reliably elicit those responses in the wild.The authors hypothesize that approximately 14,000 collected utterances provide insufficient information for some bots.
- Bots trained for questions or positive sentiment exploit those rewards successfully, but other reward relationships are less clear.The laughter-trained bot became most supportive and cheerful, while the words-elicited bot became very polite.
8.4 Interactive bot platform details
The authors built neural.chat to host dialog models for fast, real-time human interaction and evaluation. Participants converse for at least three turns, provide optional utterance-level feedback, and then rate the bot.
- neural.chat hosts deep neural network dialog models online on GPUs for fast, real-time inference and human interaction.The platform was designed to collect data from humans interacting with the bots.
- Participants can rate bots after conversations lasting at least three turns.The rating screen is accessed through the platform after the minimum conversation length.
- Annotators can optionally give feedback on individual chatbot responses using up and down arrows.This feedback is attached to specific utterances during the conversation.
- The platform kept chatbot models in one Django process and required a single Python process because PyCUDA does not support multiprocessing.All chatbots remained in memory concurrently, requiring substantial server RAM and long startup-time management.