Source-linked AI summary
Towards End-to-End Reinforcement Learning of Dialogue Agents for Information Access
Bhuwan Dhingra, Lihong Li, Xiujun Li, Jianfeng Gao, Yun-Nung Chen, Faisal Ahmed, Li Deng
TL;DR
KB-InfoBot addresses the difficulty of training dialogue agents that must query knowledge bases while preserving uncertainty and differentiability. It replaces symbolic lookup with a soft posterior over candidate entities and integrates that retrieval with reinforcement learning. The resulting Soft-KB agents achieve higher task success in fewer turns, while the fully end-to-end agent shows strong simulation learning but overfits with real users.
Problem
Hard symbolic KB lookup discards semantic-parsing uncertainty and breaks differentiability, preventing joint online training of dialogue components from user feedback.
Method
KB-InfoBot computes a posterior distribution over user-target entities from slot beliefs and supplies the full distribution to the dialogue policy through differentiable Soft-KB lookup.
Results
Soft-KB retrieval yields higher task success in fewer dialogue turns, while the end-to-end agent learns strongly in simulations but suffers overfitting with real users.
Takeaways & Limitations
Differentiable probabilistic retrieval supports reinforcement learners that discover better dialogue policies and enables end-to-end information-access agents.
Takeaways & Limitations
The model assumes independent column values, and the fully end-to-end agent overfits when tested with real users.
Abstract
from arXiv · showhide
This paper proposes KB-InfoBot -- a multi-turn dialogue agent which helps users search Knowledge Bases (KBs) without composing complicated queries. Such goal-oriented dialogue agents typically need to interact with an external database to access real-world knowledge. Previous systems achieved this by issuing a symbolic query to the KB to retrieve entries based on their attributes. However, such symbolic operations break the differentiability of the system and prevent end-to-end training of neural dialogue agents. In this paper, we address this limitation by replacing symbolic queries with an induced "soft" posterior distribution over the KB that indicates which entities the user is interested in. Integrating the soft retrieval process with a reinforcement learner leads to higher task success rate and reward in both simulations and against real users. We also present a fully neural end-to-end agent, trained entirely from user feedback, and discuss its application towards personalized dialogue agents. The source code is available at https://github.com/MiuLab/KB-InfoBot.
1 Introduction
KB-InfoBot targets multi-turn information access through a knowledge base, where conventional symbolic retrieval loses uncertainty and breaks end-to-end differentiability. The paper replaces hard lookup with soft retrieval and evaluates reinforcement-learning agents in simulation and with real users.
- Motivation: KB-InfoBots help users navigate a knowledge base to find entities through multi-turn dialogue.The paper focuses on information access rather than complicated one-shot database queries.
- Motivation: Hard-KB lookup constructs symbolic queries from semantic parsing, but discards parsing uncertainty and prevents joint parser-policy training.These properties make online end-to-end learning from user feedback difficult after deployment.
- Proposed approach: Soft-KB lookup computes a posterior over possible user targets and gives the dialogue policy the full distribution.The posterior is constructed from the agent’s beliefs about the searched entity’s attributes.
- Proposed approach: The differentiable retrieval process enables an end-to-end KB-InfoBot whose components are updated online with reinforcement learning.This directly connects retrieval, dialogue policy, and user-feedback-driven training.
- Evaluation: Evaluation uses an adapted user simulator and real users because no public benchmark exists for multi-turn KB information access.The study reports task success and dialogue efficiency for several KB-InfoBot variants.
2 Related Work
Prior work either assumes users can formulate complicated compositional queries or separates symbolic database retrieval from neural dialogue components. KB-InfoBot instead preserves modular belief tracking while replacing hard lookup with differentiable retrieval that represents uncertainty.
- Neural KB querying: Neural GenQA and neural enquirer parse complicated compositional natural-language queries into SQL-like operations.KB-InfoBot addresses interactive querying without requiring users to compose such queries.
- End-to-end dialogue: Earlier modular neural dialogue systems use Hard-KB lookup, making separate training necessary for language understanding, policy, and generation components.The symbolic retrieval step breaks differentiability across the system.
- API-based retrieval: API-based agents maintain query hypotheses outside the end-to-end system and cannot represent uncertainty beyond one slot-value at a time.This limits their treatment of uncertainty in language understanding.
- Baselines: Entropy minimization asks for the slot with maximum entropy and is optimal without language-understanding errors, serving as a baseline.The paper compares against this rule-based strategy while using an entity-centric knowledge base interface.
3 Probabilistic KB Lookup
The paper models an entity-centric knowledge base as a table and infers a posterior over user-target entities from slot-level beliefs. Soft-KB lookup combines these probabilities into a differentiable retrieval distribution while handling missing values and explicit or unknown user constraints.
- Entity-Centric Knowledge Base: An entity-centric knowledge base is represented as a table whose rows are entities, columns are attribute slots, and some entries are missing.The model assumes a domain-specific KB with one head-entity type and relation types corresponding to slots.
- Belief representation: The agent maintains slot-value distributions and probabilities indicating whether the user knows each slot’s value.These beliefs are conditioned on the user’s utterances through the current turn.
- Notations and Assumptions: The model assumes column values are independent, reducing parameters because each slot can be modeled separately rather than over KB entities directly.The paper explicitly characterizes this as a strong assumption.
- Soft-KB Lookup: For an unknown slot value, the model uses the uniform row prior; for a known value, it accounts for whether that value is missing from the KB.Missing values are treated as possible user-known values under the model’s vocabulary assumptions.
- Soft-KB Lookup: The Soft-KB lookup combines slot-level posterior probabilities to compute a posterior distribution over KB entities.The resulting procedure is obtained by combining the model’s posterior equations and missing-value handling.
4 Towards an End-to-End-KB-InfoBot
KB-InfoBot replaces symbolic database queries with a Soft-KB lookup that maintains differentiable beliefs over entities and feeds them into dialogue-policy learning. Its modular agent tracks slot beliefs, summarizes dialogue state, and selects requests or ranked KB results.
- Overview: Soft-KB lookup provides more information from language understanding and enables end-to-end training of dialogue policy and language understanding with reinforcement learning.It replaces Hard-KB retrieval with a differentiable interface between belief tracking and policy optimization.
- Overview: The agent receives a natural-language utterance and chooses either a slot-request action or an inform action returning an ordered list of KB entities.The action space contains M + 1 actions, and the dialogue ends when the agent chooses inform.
- Belief Trackers: Each slot tracker produces a distribution over slot values and a scalar probability that the user knows the slot value.The system supports hand-crafted and neural belief trackers; the neural version uses a GRU over the dialogue input history.
- Soft-KB Lookup: The Soft-KB lookup computes a posterior over database entities from slot beliefs, then supplies the posterior and entropy-based summaries to the dialogue policy.The posterior is combined with belief-tracker outputs into an internal dialogue state, which is compressed to reduce policy-network size.
- Dialogue Policy: A GRU-based neural policy uses the summarized state and dialogue history to model action probabilities, while training can sample KB entities from the posterior for inform responses.The sampled ordered results mimic a search-engine setting in which only a limited number of entities are shown.
5 Training
The agent is trained with reinforcement learning from dialogue rewards, with imitation learning used first to initialize end-to-end training. This staged procedure addresses difficult credit assignment when both actions and retrieved results may be wrong.
- Reinforcement Learning: REINFORCE trains neural components using dialogue rewards and a discounted return, with a batch-average baseline reducing update variance.The framework updates policy parameters from the reward signal and uses RMSProp for reinforcement-learning optimization.
- End-to-End Learning: End-to-end learning updates both the dialogue policy and belief trackers by treating retrieval as another policy trained from the reinforcement signal.This makes the retrieval process part of the online learning system rather than a separate symbolic operation.
- Imitation Learning: Randomly initialized end-to-end learning almost always fails for a moderately sized knowledge base.The reported difficulty arises because the agent cannot distinguish action-sequence errors from incorrect KB-result sets.
- Imitation Learning: An initial imitation-learning phase trains belief trackers and the policy to mimic hand-crafted agents before reinforcement learning.The imitation objective uses divergences between learned and rule-based belief states together with action prediction.
6 Experiments and Results
Experiments compare rule-based, reinforcement-learning, and end-to-end KB-InfoBot variants using No-KB, Hard-KB, and Soft-KB policy inputs. Soft-KB generally improves reward and task performance, while human evaluation exposes generalization limits for the fully neural agent.
- KB-InfoBot versions: The study compares Rule, RL, and E2E agents across No-KB, Hard-KB, and Soft-KB variants, with Soft-KB providing posterior-based policy inputs.The E2E agent combines neural belief tracking and policy optimization with Soft-KB lookup.
- Movies-KB: The Movies-KB uses four entity-scale splits, restricts actor and director values, and removes 20% of values from the agent’s copy.The user may still know values removed from the agent’s KB, making the task more challenging.
- Simulated User Evaluation: Soft-KB versions achieve the highest average reward, while reinforcement learning generally exceeds hand-crafted rule agents.Average reward is the optimization metric, and reward can trade off minimizing turns against maximizing success rate.
- Simulated User Evaluation: Among Soft-KB agents, E2E>RL>Rule except on the X-Large KB, where the E2E action space makes credit assignment more difficult.Both Small and X-Large settings produce lower reward because their task structure is more difficult.
- Human Evaluation: Against real users, Rule-Soft and RL-Soft outperform RL-Hard in success rate, while E2E-Soft performs comparably and takes more turns than in simulation.RL-Soft reaches 74% success with human users versus 80% against simulated users.
- Human Evaluation: The E2E agent performs best in simulation but poorly with real users, where limited simulator vocabulary contributes to weak generalization.The authors propose better language-understanding and belief-tracker architectures plus pretraining as future remedies.
7 Conclusions and Discussion
The paper advances end-to-end trainable dialogue agents for information access through differentiable probabilistic KB querying and reinforcement learning. It reports strong simulated learning for an end-to-end agent but overfitting with real users, motivating staged personalization.
- KB-InfoBot uses a differentiable probabilistic framework to query a database from beliefs over its fields or slots.
- The framework enables downstream reinforcement learning to discover better dialogue policies by providing more information.
- The end-to-end agent demonstrates strong learning capacity in simulations but suffers from overfitting when tested on real users.
- The proposed deployment strategy starts with RL-Soft and gradually switches to a personalized end-to-end agent as interaction data accumulates.
A Posterior Derivation
The posterior derivation distributes belief over entity attributes across matching and missing database values to construct a valid posterior over KB rows.
- When a slot value is missing, the derivation assumes missing values are equally likely and estimates their prior from missing-value counts.
- For a value present in the database, its belief-tracker probability mass is divided equally among all rows containing that value.
- The resulting distribution is verified to sum to 1.
B Gated Recurrent Units
The paper describes the GRU as a recurrent neural network that processes an input sequence and iteratively computes hidden states.
- At each step, the GRU computes its update gate and candidate hidden state using sigmoid and tanh transformations.
C REINFORCE updates
The REINFORCE formulation optimizes expected discounted dialogue return by differentiating trajectory likelihoods, while treating the KB posterior as part of the trainable policy when needed.
- The learner assumes access to a reward signal throughout each dialogue and optimizes expected discounted return over trajectories.
- The trajectory probability is expressed under the current policy, with the episode bounded by maximum length H and discounted by γ.
- The likelihood-ratio trick and a Markovian assumption yield the policy-gradient formulation used for reinforcement learning.
- When both policy and belief trackers are trained from reinforcement signals, the KB posterior is treated as another policy and sampled results support exploration.
- The resulting formulation modifies the episodic REINFORCE update rule, incorporating the KB posterior into training.
D Hyperparameters
The agents use distinct recurrent-state sizes and staged learning rates, with the end-to-end agent transitioning from imitation learning to reinforcement learning. Evaluation uses simulated users whose utterances can include stochastic generation and value noise.
- RL agents use GRU hidden size d = 50, while the E2E agent uses d = 100.
- Learning rates are 0.05 for imitation learning and 0.005 for reinforcement learning, with minibatch size 128.
- The E2E agent performs imitation learning for 500 updates before switching to reinforcement learning.
- Simulated users randomly select target entities and known informable slots, then respond to agent requests during multi-turn dialogues.
- User acts are rendered into template-like natural-language utterances by a separately trained sequence-to-sequence LSTM NLG with slot-value post-processing.
- User utterances may contain corrupted or substituted values and irrelevant NLG outputs; increasing softmax temperature increases this noise.