Source-linked AI summary
Key-Value Retrieval Networks for Task-Oriented Dialogue
Mihail Eric, Christopher D. Manning
TL;DR
Neural task-oriented dialogue systems struggle to incorporate knowledge-base information without explicit dialogue-state supervision. The paper introduces an end-to-end Key-Value Retrieval Network and a grounded 3,031-dialogue corpus, reporting stronger performance than competitive rule-based and neural baselines on automatic and human evaluations.
Problem
Neural task-oriented dialogue systems struggle to reason over knowledge bases while remaining end-to-end trainable and often require explicitly modeled belief trackers or dialogue state.
Method
The Key-Value Retrieval Network augments recurrent dialogue architectures with attention-based retrieval over knowledge-base key-value entries without explicit belief or intent tracker training.
Results
The model outperforms competitive heuristic and neural baselines on automatic and human evaluation metrics across grounded dialogue domains.
Takeaways & Limitations
The architecture supports grounded, multi-domain dialogue while retaining end-to-end training, and the released corpus provides a three-domain test bed for task-oriented dialogue research.
Takeaways & Limitations
Future work remains necessary for robust handling of joint knowledge-base attributes and pragmatic knowledge use such as temporal reasoning.
Abstract
from arXiv · showhide
Neural task-oriented dialogue systems often struggle to smoothly interface with a knowledge base. In this work, we seek to address this problem by proposing a new neural dialogue agent that is able to effectively sustain grounded, multi-domain discourse through a novel key-value retrieval mechanism. The model is end-to-end differentiable and does not need to explicitly model dialogue state or belief trackers. We also release a new dataset of 3,031 dialogues that are grounded through underlying knowledge bases and span three distinct tasks in the in-car personal assistant space: calendar scheduling, weather information retrieval, and point-of-interest navigation. Our architecture is simultaneously trained on data from all domains and significantly outperforms a competitive rule-based system and other existing neural dialogue architectures on the provided domains according to both automatic and human evaluation metrics.
1 Introduction
The paper targets brittle neural task-oriented dialogue systems that struggle to use knowledge bases while remaining end-to-end trainable. It introduces a domain-independent key-value retrieval architecture and a grounded three-domain dialogue corpus.
- Neural task-oriented dialogue agents struggle to reason over knowledge bases while preserving end-to-end trainability and often require explicitly annotated belief trackers or dialogue state.
- The Key-Value Retrieval Network augments recurrent dialogue models with attention-based retrieval over knowledge-base entries.It learns to extract useful knowledge-base information directly from data without explicitly training belief or intent trackers.
- The architecture is domain-independent and incorporates world knowledge into dialogue utterances through attention over key-value entries.
- 3,031 dialogues span calendar scheduling, weather information retrieval, and point-of-interest navigation in the in-car personal assistant setting.The dialogues are grounded through underlying knowledge bases.
- The model accesses database-style information while retaining neural text-generation advantages and outperforms rule-based and neural baselines on automatic and human evaluations.
2 Key-Value Retrieval Networks
The model encodes dialogue context with recurrent networks, decodes responses using attention, and retrieves knowledge-base values through key-based attention and canonicalized output tokens.
- 2.1 Encoder: Dialogue context is represented as successive user-system turns and encoded by embedding tokens into context-sensitive hidden representations with an LSTM recurrence.
- 2.2 Decoder: The decoder predicts system-response tokens recurrently and uses encoder attention at each decoding timestep.
- 2.3 Key-Value Knowledge Base Retrieval: Knowledge-base entries are normalized into subject-relation-object triples, with each knowledge base containing at most 230 triples.
- 2.3 Key-Value Knowledge Base Retrieval: At each decoding timestep, the model attends to the key of every normalized knowledge-base entry and uses the resulting logits to select its value.
- 2.3 Key-Value Knowledge Base Retrieval: Canonicalized value representations are converted at runtime into actual knowledge-base values through lookup, expanding the output vocabulary by the number of separate entries.
- 2.3 Key-Value Knowledge Base Retrieval: Aggregated knowledge-base logits are combined with original vocabulary logits, after which the argmax token is selected for the next decoding step.
3 A Multi-Turn, Multi-Domain Dialogue Dataset
The paper introduces a multi-turn corpus spanning three in-car assistant domains, collected through Wizard-of-Oz interactions grounded in private knowledge bases. Its design varies tasks, slot specifications, and dialogue conditions to support naturalistic and generalizable dialogue modeling.
- The corpus contains multi-turn dialogues in calendar scheduling, weather information retrieval, and point-of-interest navigation.
- Data Collection: Data collection used a Wizard-of-Oz scheme with Driver and Car Assistant roles, where the assistant accessed a private knowledge base.
- Data Collection: Tasks randomly selected values for three to five domain-specific slots from per-domain candidate sets.
- Data Collection: Private knowledge bases contained six to seven rows and five to seven attribute types, with some attributes intentionally omitted.
- Data Collection: Roughly half of scheduling dialogues did not explicitly require a knowledge base, instead encouraging clarification for underspecified tasks.
- Data Collection: The corpus also included user-recorded commands prompted by underspecified visual depictions to encourage more naturalistic utterances.
- Data Collection: 241 unique Amazon Mechanical Turk workers collected the data over approximately six days.
4 Related Work
Prior task-oriented dialogue research includes symbolic planning, neural state and belief tracking, reinforcement learning, and knowledge-aware models. The paper positions its approach against brittleness, computational cost, explicit supervision, and the limited availability of public datasets.
- POMDP and reinforcement-learning approaches use carefully designed action spaces but can become brittle and computationally intractable as action states proliferate.
- Neural dialogue research has applied distributed representations to dialogue state management, belief tracking, and response generation.
- Some task-oriented models support supervised and reinforcement learning while incorporating domain knowledge through explicit features and output restrictions.
- The paper contrasts its strictly supervised token-generation process with approaches relying on dialogue state trackers.
- Task-oriented dialogue research has often lacked standard, publicly available datasets, with prior corpora commonly tied to travel booking or dialogue-state tracking.
5 Experiments
The experiments compare the Key-Value Retrieval Network with rule-based and neural baselines using automatic and human evaluation. The proposed model achieves the strongest automatic results and outperforms baselines in real-time human evaluation, while human performance remains an upper bound.
- Experimental setup: The experiments report automatic and human evaluations of the proposed model alongside rule-based and neural baselines.The automatic evaluation includes BLEU and entity F1; human evaluation assesses correctness, appropriateness, and humanlikeness.
- Automatic evaluation: Table 3 evaluates aggregate and domain-specific entity F1, including scheduling, weather, and navigation results.The table compares Attn. Seq2Seq, KV Retrieval Net without encoder attention, and other models.
- Baseline results: Copy Net has the lowest aggregate entity F1 but the second-highest BLEU score, reflecting stronger language modeling than cross-domain entity integration.Its entity F1 is highest for weather but poor in the other domains.
- Baseline results: The rule-based model has the lowest BLEU score but competitive aggregate entity F1 because manually provided templates limit naturalness while heuristics support KB querying.Its design explicitly parses user semantics and queries the underlying knowledge base.
- Automatic evaluation: The Key-Value Retrieval Network achieves the highest BLEU, aggregate entity F1, and scheduling and navigation entity F1, outperforming rule-based aggregate entity F1 by 4.2%.It also exceeds Copy Net by 2.2 BLEU points and 11% in entity F1 without heuristics or manual labeling.
- Human evaluation: Human performance exceeds the KV network’s entity F1 by approximately 12.7%, while the model is on par with humans in BLEU.In real-time dialogues, the KV network outperforms the baseline models on all reported human-evaluation metrics.
- Human evaluation: Sample dialogues show the model integrating knowledge-base information into naturalistic utterances across multiple domains.The examples come from human evaluation dialogues between Turkers and the KV Retrieval Net.
6 Conclusion and Future Work
The work presents a neural task-oriented dialogue model that retrieves knowledge-base information to sustain grounded discourse across domains and generate natural-sounding responses without explicitly modeling dialogue state. It also contributes a public three-domain dataset, while identifying remaining gaps in matching human performance and handling complex knowledge-base use.
- Conclusion: The model retrieves world knowledge from knowledge bases to sustain grounded discourse across varied domains.Its architecture incorporates this knowledge into dialogue responses in an end-to-end trainable fashion.
- Conclusion: It incorporates knowledge into natural-sounding responses without explicitly modeling dialogue state.
- Conclusion: The authors introduce a publicly available dialogue dataset spanning three in-car personal assistant domains.
- Future Work: Future work targets the remaining margin between the Key-Value Retrieval Network and human performance.Planned directions include robust handling of joint knowledge-base attributes and pragmatic reasoning such as temporal reasoning.