Source-linked AI summary
Mem2Seq: Effectively Incorporating Knowledge Bases into End-to-End Task-Oriented Dialog Systems
Andrea Madotto, Chien-Sheng Wu, Pascale Fung
TL;DR
Task-oriented dialog systems need to incorporate external Knowledge Bases, but existing approaches struggle with KB integration and complex or time-consuming processing. Mem2Seq combines multi-hop memory attention with pointer-based sequential generation, achieving fast training and state-of-the-art results across three task-oriented dialog datasets. The paper also analyzes how the model accesses memories and uses vocabulary generation.
Problem
Existing task-oriented dialog approaches struggle to incorporate external Knowledge Base information into RNN hidden states, while long-sequence attention processing is time-consuming.
Method
Mem2Seq combines a multi-hop memory-network encoder and pointer-based sequential decoder to copy words from dialog history or Knowledge Bases during generation.
Results
Mem2Seq is fast and achieves state-of-the-art results on three different task-oriented dialog datasets.
Takeaways & Limitations
Mem2Seq can generate relevant answers using both external KB information and a predefined vocabulary while learning correlations between memories through multi-hop attention.
Takeaways & Limitations
The KV Retrieval Net baseline uses simplified entity normalization and linking, making its reported evaluation not directly comparable with Mem2Seq.
Abstract
from arXiv · showhide
End-to-end task-oriented dialog systems usually suffer from the challenge of incorporating knowledge bases. In this paper, we propose a novel yet simple end-to-end differentiable model called memory-to-sequence (Mem2Seq) to address this issue. Mem2Seq is the first neural generative model that combines the multi-hop attention over memories with the idea of pointer network. We empirically show how Mem2Seq controls each generation step, and how its multi-hop attention mechanism helps in learning correlations between memories. In addition, our model is quite general without complicated task-specific designs. As a result, we show that Mem2Seq can be trained faster and attain the state-of-the-art performance on three different task-oriented dialog datasets.
1 Introduction
Task-oriented dialog systems need external Knowledge Bases, but pipelined architectures require complex module dependencies and human effort, while end-to-end models struggle to incorporate KB information efficiently. Mem2Seq addresses these challenges with multi-hop memory attention and pointer-based generation.
- External KB queries are essential because task-oriented responses depend on both dialog history and query results.
- Pipelined dialog systems make module dependencies complex and require human effort for KB interpretation.
- End-to-end RNN approaches reduce the need for handcrafted dialog-state labels and can copy words from input sources.
- End-to-end approaches struggle to incorporate external KB information into RNN hidden states, and long-sequence attention processing is time-consuming.
- MemNNs efficiently read external memories and encode long dialog histories, but traditionally select responses from a predefined candidate pool.
- Mem2Seq combines multi-hop attention with pointer networks to generate responses while copying words from dialog history or KBs.
- Mem2Seq is reported to support dynamic memory-access queries, faster training, and state-of-the-art results across several task-oriented dialog datasets.
2 Model Description
Mem2Seq combines a MemNN encoder with a memory decoder that generates responses by jointly using vocabulary words and pointers to dialog-history or KB memories. Multi-hop attention, dynamic queries, and a sentinel gate control memory reading and token generation.
- Architecture: Mem2Seq combines a MemNN encoder with a memory decoder that reads and copies from dialog history and KB memories.The encoder represents the dialog history, while the decoder reads memory contents to generate responses.
- Memory Decoder: A sentinel token acts as a hard gate: generated tokens come from the vocabulary when the expected word is absent from memory, otherwise from the pointer distribution.The pointer distribution is trained to select the sentinel when the target word does not appear in memory.
- Memory Encoder: The encoder performs K attention hops, updates its query with each readout, and passes the final memory vector to decoding.At hop k, attention selects relevant memories and the query is updated as q_k+1 = q_k + o_k.
- Memory Decoder: The decoder produces both a vocabulary distribution and a pointer distribution over dialog-history and KB memory contents at every step.The vocabulary distribution uses the first-hop readout and current query, while the pointer distribution uses the final decoder hop.
- Training: Mem2Seq jointly trains vocabulary and pointer behavior with the sum of two cross-entropy losses, while first- and last-hop attention serve looser retrieval and sharper token selection roles.The architecture expects the first hop to retrieve memory information and the final hop to select an exact token using pointer supervision.
- Memory Content: The model stores word-level dialog tokens with temporal and speaker information and represents KB entries as subject-relation-object memories.KB tuple embeddings are formed by summing embeddings of the subject, relation, and object; the object is used as the pointed-to generated word.
3 Experimental Setup
The experiments evaluate Mem2Seq on three task-oriented dialog datasets using accuracy, BLEU, and entity F1, with comparisons against memory, recurrent, and pointer-based baselines. Results report strong performance across simulated, human-bot, and multi-domain human-human settings, while some external evaluations are not directly comparable.
- Datasets: Mem2Seq is evaluated on bAbI Dialog, DSTC2, and In-Car Assistant, covering simulated restaurant dialogs, noisy human-bot dialogs, and diverse multi-domain conversations.The In-Car Assistant dataset includes calendar scheduling, weather retrieval, and point-of-interest navigation with more complicated KB information.
- Experimental Configuration: The experiments compare Mem2Seq with QRN, MemNN, GMemNN, Seq2Seq variants, and Ptr-Unk using one, three, and six hops.Training uses Adam, greedy decoding, and hyperparameter ranges for learning rate, embedding size, and dropout.
- Evaluation Metrics: Per-response accuracy requires an exact response match, whereas per-dialog accuracy requires every generated response in the dialog to be correct.Because Mem2Seq generates tokens individually, this evaluation is more challenging than selecting from predefined response candidates.
- Evaluation Metrics: Entity F1 is micro-averaged over system responses using plain-text entities rather than canonicalized forms or slot-tracking accuracy.The authors use entity F1 because the datasets are not designed for slot tracking.
4 Experimental Results
Mem2Seq achieves strong results across bAbI Dialog, DSTC2, and In-Car Assistant while retaining advantages on out-of-vocabulary data and longer knowledge-base inputs. It also trains substantially faster than attention-based Seq2Seq models as input length grows.
- bAbI Dialog: Mem2Seq with six hops achieves 97.9% per-response and 69.6% per-dialog accuracy on T5, versus 84.5% and 2.3% on T5-OOV.The authors report that these results surpass existing methods and that OOV performance incurs comparatively little loss.
- DSTC2: Mem2Seq reaches 75.3% entity F1 and 55.3 BLEU on DSTC2, while per-response accuracy remains within a 2% margin of other solutions.The dataset is noisy, and every model’s exact-match per-response accuracy is below 50%.
- In-Car Assistant: Mem2Seq obtains the highest 12.6 BLEU score on In-Car Assistant and an entity F1 score of 33.4%, generally exceeding the baselines.The authors note that some reported baseline numbers are not directly comparable.
- Interpretation: Human responses can have high entity F1 but low BLEU, suggesting that correct entity selection may matter even when wording differs from the reference.The authors therefore view Mem2Seq as showing potential for successfully choosing correct entities.
- Training Efficiency: Mem2Seq is around 5 times faster than attention-based Seq2Seq on In-Car Assistant and DSTC2, with the gap increasing as maximal input length grows.The authors attribute this efficiency difference to Seq2Seq input dependencies and the need to encode KBs, whereas Mem2Seq encodes dialog history.
5 Analysis and Discussion
Mem2Seq’s analyses show how multi-hop attention retrieves and generates memory-grounded responses while using vocabulary generation when needed. The model’s hops, visualized attention, and examples illustrate focused retrieval, query specialization, and improved response correctness.
- Memory Attention: Mem2Seq selects relevant memory tokens with sharp last-hop attention, retrieving KB addresses while using the vocabulary when a word is absent from memory.The model retrieved “270 altarie walk” at the sixth step and generated “is” from the vocabulary after triggering the sentinel.
- Multiple Hops: The first hop generally scores and retrieves relevant memories, whereas the last hop focuses on a specific token and can make mistakes when attention is not sharp.The analysis attributes some mistakes to a lack of hops in particular samples.
- Query Vectors: Query-vector separation becomes clearer from hop 1 to hop 6, with later-hop clusters corresponding to categories such as location, cuisine, and number.The analysis associates early hops with retrieving more information and later hops with pointing into memories.
- Discussions: Mem2Seq is presented as a simple generative model that incorporates KB information with promising generalization ability and requires less human interference than conventional systems.The discussion also argues that entity F1 may be more comprehensive than per-response accuracy or BLEU for varied but correct responses.
6 Related Works
Related work spans recurrent end-to-end dialog models, memory-based decoders, and non-recurrent architectures. Mem2Seq differs by combining multi-hop attention with a copy mechanism for KB-aware generation.
- Memory Networks: End-to-end memory networks repeatedly read external memories with query vectors, but their response selection is limited by predefined candidate pools.Mem2Seq extends the memory-network approach with sequential generation.
- Sequence-to-Sequence Models: Seq2Seq models improve language modeling but perform poorly at KB retrieval, while copy-augmented variants address copying entities from inputs.Even sophisticated attention models may fail to map correct entities into generated responses.
- Memory-Based and Non-Recurrent Models: Mem2Seq’s query-generation phase relates to memory controllers and memory encoders, but its multi-hop attention and copying distinguish it from models using a single matrix representation.The comparison also places Mem2Seq alongside non-recurrent self-attention-based generative models.
7 Conclusion
Mem2Seq is an end-to-end trainable model that combines memory-network multi-hop attention with pointer-style copying for external information. The paper reports relevant answers using KBs and vocabulary, learned memory correlations, fast training, generality, and state-of-the-art results on three datasets.
- Mem2Seq combines multi-hop attention from end-to-end memory networks with pointer networks to incorporate external information.
- The model produces relevant answers using both external KB information and a predefined vocabulary while visualizing learned correlations between memories.
- Mem2Seq is reported as fast, general, and state-of-the-art on three different task-oriented dialog datasets.
8 Tables
The tables summarize training speed and selected hyper-parameters across datasets and hop settings. Mem2Seq is identified as faster especially for longer inputs, while the hyper-parameter table specifies embedding, GRU, dropout, and learning-rate settings.
- Training Time: Mem2Seq is faster than other models, especially for longer inputs, according to the minutes-per-epoch table.
- Hyper-parameters: The hyper-parameter table reports embedding dimensions, GRU hidden sizes, dropout rates, and a learning rate of 0.001 with decay rate 0.5.
9 Visualization
The visualizations show multi-hop attention becoming more focused across hops and associate a non-sharp final hop with mistakes. Additional examples compare correct and generated responses for bAbI and In-Car settings.
- Multi-hop attention: At generation step 8, attention is shallow and evenly divided across five choices at hop 1, then becomes sharp at hop 3.The visualization is from bAbI dialog task 3.
- Multi-hop attention: The model tends to make mistakes when attention at the last hop is not sharp.
- Response comparisons: One example shows identical correct and generated API-call responses: “api call italian london eight expensive.”
- Response comparisons: The visualizations compare correct responses, labeled COR, with generated responses, labeled GEN, for bAbI and In-Car datasets.
- Response comparisons: Another generated response preserves the parking-garage facts but changes their ordering and adds “through the directions.”
10 InCar Assistant Dataset Examples
The examples cover generated responses for the In-Car Assistant across weather, navigation, and scheduling domains. A scheduling example contrasts Seq2seq, PTRUNK, and Mem2Seq outputs for a reminder request.
- Weather: Tables 9 and 10 present example generated answers for the In-Car dataset’s weather domain.
- Navigation: Tables 11–14 present example generated responses for the In-Car Assistant’s navigation domain.
- Scheduling: Table 15 compares Seq2seq, PTRUNK, and Mem2Seq on a scheduling reminder request involving football activity, 1pm, the 11th, and Marie.
- Scheduling: Table 16 provides another example of generated responses for the In-Car Assistant’s scheduling domain.