Source-linked AI summary
Evaluating Prerequisite Qualities for Learning End-to-End Dialog Systems
Jesse Dodge, Andreea Gane, Xiang Zhang, Antoine Bordes, Sumit Chopra, Alexander Miller, Arthur Szlam, Jason Weston
TL;DR
End-to-end dialog models lacked scalable evaluation of goal-oriented abilities beyond language quality. The paper introduces four large movie-domain tasks using real sources and automatic ranking evaluation, finding that some neural models perform reasonably across tasks while Memory Networks exploit context and knowledge bases, though limitations remain in recommendation labeling and joint training.
Problem
End-to-end dialog systems lack scalable evaluation paradigms that assess goal-oriented dialog rather than only generated-language quality.
Method
The paper constructs four automatically evaluated movie-domain tasks spanning QA, recommendation, combined multi-turn dialog, and Reddit response selection.
Results
Some end-to-end neural models perform reasonably across all tasks, with Memory Networks using short- and long-term memory to leverage context and factual knowledge.
Takeaways & Limitations
Strong end-to-end dialog systems should perform across objective, personalized, contextual, and chit-chat tasks, a necessary but not sufficient condition for full functionality.
Takeaways & Limitations
Recommendation hits@k underestimates performance because incomplete labeling omits potentially good predictions, and the dataset is focused on movies.
Abstract
from arXiv · showhide
A long-term goal of machine learning is to build intelligent conversational agents. One recent popular approach is to train end-to-end models on a large amount of real dialog transcripts between humans (Sordoni et al., 2015; Vinyals & Le, 2015; Shang et al., 2015). However, this approach leaves many questions unanswered as an understanding of the precise successes and shortcomings of each model is hard to assess. A contrasting recent proposal are the bAbI tasks (Weston et al., 2015b) which are synthetic data that measure the ability of learning machines at various reasoning tasks over toy language. Unfortunately, those tests are very small and hence may encourage methods that do not scale. In this work, we propose a suite of new tasks of a much larger scale that attempt to bridge the gap between the two regimes. Choosing the domain of movies, we provide tasks that test the ability of models to answer factual questions (utilizing OMDB), provide personalization (utilizing MovieLens), carry short conversations about the two, and finally to perform on natural dialogs from Reddit. We provide a dataset covering 75k movie entities and with 3.5M training examples. We present results of various models on these tasks, and evaluate their performance.
1 INTRODUCTION
The paper addresses weak, poorly scalable evaluation of end-to-end dialog systems by proposing large, automatically evaluated movie-domain tasks that test prerequisite capabilities.
- End-to-end dialog systems directly generate responses from user utterances and dialog context without explicit dialog-state tracking.
- Existing evaluations rely on few human scores, crowdsourcing, or BLEU, which assess language quality but not whether systems conduct dialog effectively.
- The paper proposes four movie-domain tasks: question answering, recommendation, combined recommendation and QA, and Reddit-style chit-chat.
- The dataset uses OMDb, MovieLens, and Reddit, covering ∼75k movie entities and ∼3.5M training examples.
2 THE MOVIE DIALOG DATASET
The movie dialog dataset evaluates factual answering, personalization, contextual recommendation conversations, and natural Reddit response selection using large-scale, rankable examples.
- The dataset defines four tasks covering factual QA, personalized movie recommendation, multi-turn QA plus recommendation, and Reddit chit-chat.
- 2.1 QUESTION ANSWERING (QA): The QA task builds a knowledge base from OMDb and MovieLens metadata, with 11 question classes spanning relations such as actors, directors, genres, years, and languages.
- 2.1 QUESTION ANSWERING (QA): QA evaluation ranks candidate answers and reports hits@1, measuring whether the top-ranked answer is correct.
- 2.2 RECOMMENDATION DATASET: Recommendation dialogs sample users’ five-star movies as preferences and select another five-star movie from the same user as the target answer.
- 2.2 RECOMMENDATION DATASET: Recommendation hits@k values are lower-bound estimates because incomplete ratings labels can omit good predictions that lack the single recorded answer.
- 2.3 QA+RECOMMENDATION DIALOG: The combined task uses three exchanges integrating recommendation, a factoid question about the suggestion, and an alternative recommendation informed by stated tastes.
- 2.4 REDDIT DISCUSSION: The Reddit task flattens multi-participant movie discussions into parent-comment dialogs and ranks each true response among 10,001 candidates.
3 MODELS
The paper evaluates several end-to-end dialog model families, emphasizing Memory Networks that combine conversational context with retrieved long-term knowledge. It also compares supervised embeddings, LSTMs, question-answering systems, SVD, and information-retrieval baselines.
- 3.1 MEMORY NETWORKS: Memory Networks augment dialog context with long-term knowledge retrieved by hashing words from recent messages into a movie knowledge base.Retrieved sentences and recent conversational messages jointly form the model memory.
- 3.1 MEMORY NETWORKS: Memory Networks embed the current input and memories, compute softmax attention from inner products, and combine the weighted output with the input.The attention procedure can be stacked into multiple hops.
- 3.1 MEMORY NETWORKS: The final prediction ranks candidate responses using q⊤Wy_c scores, with vocabulary words for Tasks 1–3 and target sentences for Task 4.Training minimizes cross-entropy between the predicted distribution and the true label.
- 3.2 SUPERVISED EMBEDDING MODELS: Supervised embedding models independently sum input and target word embeddings, compare them with a similarity metric, and train with a ranking loss.The paper notes these methods may struggle on question answering because memorization must reside in individual word embeddings.
- 3.3 RECURRENT NEURAL NETWORKS: Standard LSTMs serve as the baseline because evaluating all RNN variants is beyond the work’s scope; attention-based LSTMs share properties with Memory Networks.The paper expects LSTMs to be challenging on question answering and recommendation tasks.
- 3.4–3.6 BASELINES: Task-specific baselines include an embedding-based database-ranking QA system, SVD for recommendation, and nearest-neighbour information retrieval for candidate responses.SVD requires a user × item matrix and discards the textual input used by end-to-end dialog models.
4 RESULTS
Across the movie-dialog tasks, model strengths depend on the required capability: explicit knowledge-base access helps factual question answering, while memory and context help broader dialogue tasks. Joint training preserves Memory Networks’ performance relatively well but degrades Supervised Embeddings.
- Answering Factual Questions: Memory Networks and the baseline QA system outperform other methods on factual questions because they explicitly access the knowledge base.LSTMs perform particularly poorly on this task.
- Answering Factual Questions: The task-specific baseline QA system is superior to Memory Networks, while Memory Networks’ general design performs well on other dialogue tasks.This comparison indicates room for improvement in Memory Networks but highlights their broader applicability.
- Making Recommendations: LSTMs, Supervised Embeddings, and Memory Networks perform similarly on recommendations, and all outperform the SVD baseline.The authors conjecture that recommendations resemble language modeling because the input is a sequence of similar recommendations.
- Using Dialog History: Memory Networks outperform Supervised Embeddings on QA+Recommendations and Reddit because they use dialogue context more effectively.The advantage appears in later responses and comes from short-term dialogue history rather than knowledge-base access.
- Joint Learning: Joint training makes Supervised Embeddings perform more poorly, whereas Memory Networks incur only slight losses on Tasks 2–4 and a slight gain on Task 1.Memory Networks can distribute modeling capacity across word embeddings and attention over long- and short-term memory.
5 UBUNTU DIALOGUE CORPUS RESULTS
The Ubuntu Dialog Corpus provides an external retrieval-based evaluation on longer technical dialogues. Increasing memory hops in MemN2N improves performance substantially over the prior best reported model.
- Dataset: The Ubuntu Dialog Corpus contains almost 1M multi-turn dialogues and 100M words from Ubuntu IRC conversations.It includes 900k training, 20k validation, and 20k test dialogues, averaging more than seven turns.
- Evaluation: Ubuntu evaluation retrieves the correct response from 10 candidates and reports Hits@1.The setup is similar to Reddit Task 4.
- Results: More than 2 memory hops increase MemN2N performance by over +8% relative to the previous best reported model.The 1-hop model performs similarly to the LSTM, while additional hops continue improving over 1 hop but add little beyond 2 hops.
6 CONCLUSION
The paper introduces a broad benchmark for evaluating end-to-end dialogue skills across goal-driven and open-ended movie conversations. Results show promising general performance for memory-based models, but factual QA and joint training remain limitations, while the task design can transfer beyond movies.
- Conclusion: The benchmark evaluates factual question answering, personalized recommendation, mixed short dialogue, and Reddit-style chit-chat.Together, these tasks cover objective and subjective goal-driven dialogue as well as less goal-driven conversation.
- Conclusion: Some end-to-end neural models perform reasonably across all tasks compared with standard per-task baselines.Memory Networks use short- and long-term memory to exploit local context and factual knowledge bases.
- Conclusion: Memory Networks remain weaker than stand-alone QA systems, and performance degrades when all four tasks are trained jointly.The paper identifies overcoming both problems as future work.
- Conclusion: Although the dataset focuses on movies, the task design can be transferred to domains such as sports, music, and restaurants.Future datasets should test whether models are overtuned to these goals and evaluate additional skills.
A FURTHER EXPERIMENTAL DETAILS
The experiments compare entity-aware dictionaries, memory networks, supervised embedding models, LSTMs, and relevance-feedback retrieval across the dialog tasks. Model choices and retrieval behavior vary substantially by task, with LSTMs performing poorly on Reddit response selection and relevance feedback degrading performance.
- Dictionary: Entity names are represented as single dictionary elements, allowing QA and recommendation models to predict entities directly.The remaining vocabulary consists of unigrams not covered by the entity dictionary.
- Memory Networks: Memory Networks usually use one hop, but Task 3 performs better with two or three hops than with one.The joint task also increases model capacity through separate input, memory, and output dictionaries and uses pretraining without long-term memory.
- Supervised Embedding Models: Supervised embedding models use either shared embeddings or separate input-context and label embeddings, with the better variant depending on the task.The single-dictionary model works better on Tasks 1 and 4, while the two-dictionary model works better on another task described in the experiments.
- LSTMs: LSTMs perform poorly on Reddit response selection despite reasonable perplexity, partly because response lengths create unbalanced sequence probabilities.The resulting procedure is also much slower because candidate responses cannot be embedded once and cached.
- Information Retrieval: Increasing the relevance-feedback weight degrades the hybrid retrieval model, including at a weight of 0.5.The method retrieves a similar historical message, adds its response to the query, and scores candidate responses using the combined input.
B OPTIMAL HYPER-PARAMETER VALUES
Hyperparameters are selected by grid search on validation data, with task-specific settings for embedding dimension, learning rate, dictionaries, memory hops, and LSTM unfolding depth. Joint-task settings optimize normalized mean performance across the four tasks.
- Selection procedure: Grid search on the validation set selects embedding dimension d, learning rate λ, number of dictionaries w, memory hops K, and LSTM unfolding depth blen.All learning models are implemented in Torch.
- Task-specific settings: Task-specific configurations include QA-system λ = 0.001, d = 50 and SVD d = 50.These are listed among the optimal hyperparameter values for the task configurations.
- Task-specific settings: The supervised embedding model uses λ = 0.05, d = 50, w = 1 in one configuration, while another uses λ = 0.005, d = 200, w = 2.The corresponding MemN2N settings are λ = 0.005, d = 50, w = 1, K = 1 and λ = 0.01, d = 1000, w = 1, K = 1.
- Task 3: For Task 3, MemN2N uses λ = 0.001, d = 50, w = 1, K = 3, while LSTM uses λ = 0.001, d = 100, blen = 10.The section identifies Task 3 as QA+Recommendation.
- Joint task: Joint-task hyperparameters maximize mean performance over four tasks after scaling each task by its best development-set model.This scaling normalizes the metrics before averaging.
- Ubuntu Dialog Corpus: For Ubuntu Dialog Corpus, MemN2N settings are selected by validation-set grid search, with λ = 0.001 and d = 256 reported for the best models with K = 1, 2, 3, 4.The reported models vary the number of memory hops.
C.1 BREAKDOWN OF TASK 1 (QA) RESULTS BY QUESTION TYPE
Table 8 reports QA test performance broken down by question type using hits-at-one and hits-at-ten metrics.
- Table scope: The table organizes QA test results by question type.
- Metrics: Performance is reported with the h@1 metric.
- Metrics: Performance is also reported with the h@10 metric.
C.2 BREAKDOWN OF TASK 3 (QA+RECOMMENDATION) RESULTS BY RESPONSE TYPE
Table 9 reports QA+Recommendation test results using the hits-at-ten metric and includes a MemN2N comparison without long-term knowledge-base memory.
- Table scope: The table reports test results for the QA+Recommendation task.
- Metric: Results are evaluated with the h@10 metric.
- Memory comparison: The final row compares MemN2N without access to long-term memory from the knowledge base.
C.3 BREAKDOWN OF TASK 4 (REDDIT) RESULTS BY RESPONSE TYPE
Table 10 reports Reddit-task test results using the h@10 metric and includes MEMN2N without knowledge-base access as a comparison model.
- The table reports test results for the Reddit task.
- Results are evaluated with the h@10 metric.
- MEMN2N (-KB) denotes a Memory Network model without access to the knowledge base.