Source-linked AI summary
Learning End-to-End Goal-Oriented Dialog
Antoine Bordes, Y-Lan Boureau, Jason Weston
TL;DR
Goal-oriented dialog systems traditionally depend on handcrafted domain structures, raising questions about whether end-to-end methods transfer beyond chit-chat. This paper introduces a restaurant-reservation testbed that decomposes dialog into API and knowledge-base operations, finding that Memory Networks are promising but still unreliable for complete dialogs.
Problem
Handcrafted slot-filling systems are reliable but hard to scale, while chit-chat results do not establish effectiveness for goal-oriented dialog.
Method
The paper builds an open restaurant-reservation testbed with decomposed tasks, simulated knowledge-base dialogs, and evaluations on DSTC2 and concierge conversations.
Results
Memory Networks outperform a dedicated slot-filling baseline and solve API issuance and updating adequately, but per-dialog accuracy remains close to 0 on Tasks 3 and 4.
Takeaways & Limitations
The testbed supports reproducible, interpretable evaluation of end-to-end goal-oriented dialog while exposing unresolved weaknesses in using API outputs and completing dialogs.
Takeaways & Limitations
The controlled tasks are partly artificial, and real datasets introduce noisier, more variable language and dialog behavior.
Abstract
from arXiv · showhide
Traditional dialog systems used in goal-oriented applications require a lot of domain-specific handcrafting, which hinders scaling up to new domains. End-to-end dialog systems, in which all components are trained from the dialogs themselves, escape this limitation. But the encouraging success recently obtained in chit-chat dialog may not carry over to goal-oriented settings. This paper proposes a testbed to break down the strengths and shortcomings of end-to-end dialog systems in goal-oriented applications. Set in the context of restaurant reservation, our tasks require manipulating sentences and symbols, so as to properly conduct conversations, issue API calls and use the outputs of such calls. We show that an end-to-end dialog system based on Memory Networks can reach promising, yet imperfect, performance and learn to perform non-trivial operations. We confirm those results by comparing our system to a hand-crafted slot-filling baseline on data from the second Dialog State Tracking Challenge (Henderson et al., 2014a). We show similar result patterns on data extracted from an online concierge service.
1 INTRODUCTION
The paper introduces a reproducible testbed for evaluating end-to-end systems on goal-oriented restaurant dialogs, where models must manipulate language, symbols, API calls, and API outputs. Memory Networks show encouraging but imperfect performance, including outperforming a handcrafted baseline while remaining unreliable at completing full dialogs.
- Traditional slot-filling systems reliably track predefined dialog slots but are difficult to scale because users may refer to unanticipated features.Restaurant slots include location, price range, and cuisine type.
- End-to-end systems train all components directly from dialogs without assuming a fixed domain or dialog-state structure, but chit-chat success may not transfer to goal-oriented conversations.
- The paper proposes an open, lightweight resource that decomposes goal-directed dialog into subtasks for reproducible comparison and built-in error analysis.
- Five simulated restaurant-reservation tasks test dialog management, API-call manipulation, use of knowledge-base outputs, and complete conversations.
- The evaluation includes simulator-generated data, converted DSTC2 dialogs, and online-concierge conversations to examine whether task results reflect real interactions.
- Memory Networks outperform a dedicated slot-filling baseline on DSTC2, while per-response performance is encouraging but per-dialog accuracy remains low.
2 RELATED WORK
Prior goal-oriented systems rely on structured state and action representations with substantial handcrafted features, while existing end-to-end resources are scarce or unsuitable for full dialog learning. The paper positions its testbed as a more open and dialog-complete evaluation resource.
- POMDP-based goal-oriented systems model conversation successfully but still require handcrafted features for state and action representations, restricting them to narrow domains.
- Few suitable open resources exist for training and testing end-to-end goal-oriented dialog models because many datasets target state tracking, are noisy, small, proprietary, or restricted.
- The closest related tasks are question-answering benchmarks rather than full dialogs because their bots provide answers but never ask questions.
3 GOAL-ORIENTED DIALOG TASKS
The paper constructs restaurant-reservation dialogs from a structured knowledge base and decomposes them into tasks requiring state tracking, API interaction, result use, and generalization to unseen entities. It supplements controlled simulation with DSTC2 and noisier, more variable concierge dialogs.
- Restaurant reservation simulation: The simulator represents restaurants with cuisine, location, price range, rating, party-size availability, address, and phone number, queried through four-field API calls.
- Restaurant reservation simulation: Simulation dialogs combine user and bot utterances with API calls and returned facts, using 43 user patterns and 20 bot patterns to create varied utterances.
- Task definitions: Tasks 1 and 2 test implicit dialog-state tracking by issuing and updating API calls, while Tasks 3 and 4 test using knowledge-base facts and Task 5 combines all tasks.
- Task definitions: Task 3 requires listing restaurants in descending rating order until acceptance, whereas Task 4 requires answering requests for phone numbers, addresses, or both.
- Datasets: Out-of-vocabulary test sets use disjoint restaurants, locations, cuisines, phones, and addresses to test generalization to entities absent from training dialogs.
- Datasets: The datasets contain 1,000-example training sets and evaluate candidate ranking for bot utterances and API calls rather than free-form generation.
- DSTC2: DSTC2 dialogs are harder than the simulator because speech errors, misinterpretations, and nondeterministic bot behavior introduce noise.
4 MODELS
The paper evaluates rule-based, information-retrieval, supervised-embedding, and Memory Network approaches for end-to-end goal-oriented dialog. These models address response selection, conversation history, memory-based reasoning, and entity matching, including out-of-vocabulary entities.
- The evaluation compares rule-based systems, classical information retrieval, supervised embeddings, and end-to-end Memory Networks.
- Classical information retrieval models: TF-IDF Match ranks candidate responses by TF-IDF-weighted cosine similarity using either the last utterance or the entire conversation history.
- Classical information retrieval models: Nearest Neighbor retrieves the most similar training conversation using word overlap and returns its associated response.
- Supervised embeddings: Supervised embedding models learn response-scoring mappings from dialogs with a margin-ranking objective and sampled negative responses.
- Memory Networks: Memory Networks iteratively read stored dialog history and short-term context, using hops to reason about the required response.
- Memory Networks: Match type features target entities by supporting exact matching and handling out-of-vocabulary words through seven knowledge-base entity-type markers.
5 EXPERIMENTS
Across simulated and realistic restaurant-dialog tasks, Memory Networks generally outperform the other learned baselines, but reliable goal completion remains difficult. Performance improves on several subtasks and with match types, while API-result interpretation remains a central weakness.
- TF-IDF Match performs worst on both simulated and real data, remaining far behind Nearest Neighbor even after adding match type features.
- Supervised embeddings outperform classical information retrieval generally, but achieve complete success only on T1, while mean dialog accuracy is 0 on the other tasks.
- Memory Networks without match type features outperform classical information retrieval and supervised embeddings across all evaluated tasks.
- Memory Networks adequately solve issuing and updating API calls, but per-dialog accuracy remains close to 0 on T3 and T4.
- Adding match type features makes T4 solvable and significantly improves out-of-vocabulary results, while T3 and T5 remain failure cases.
- Perfectly coded rule-based systems solve simulated T1-T5 perfectly, whereas Memory Networks outperform the rule-based system on the more realistic T6 task.
- The same relative performance pattern appears on T6 and Concierge, suggesting that simulated-task analysis can serve as an evaluation proxy for realistic data.
6 CONCLUSION
The paper introduces an open, systematic testbed for evaluating end-to-end goal-oriented dialog learning. Memory Networks are effective relative to the baselines but remain deficient in key capabilities.
- The paper introduces an open dataset and task set for systematic, controlled evaluation of end-to-end goal-oriented dialog learning methods.
- The task breakdown is intended to improve reproducibility, focus research and development, and support evaluation of dialog systems with practical utility.
- Memory Networks are effective relative to the evaluated baselines but still lack some key goal-oriented dialog capabilities.
A MEMORY NETWORKS IMPLEMENTATION
The model stores the conversation in memory, attends to relevant history, iteratively updates its controller state, and selects the next utterance or API call from candidates.
- At each time step, previous user utterances and model responses are appended to memory, representing the conversation history.The model retains the latest user utterance as the controller input while storing earlier exchanges in memory.
- Training uses complete dialog transcripts, with the upcoming model response serving as the target.
- Each utterance is represented as a bag-of-words vector using an embedding matrix, with time and speaker features added to memory representations.The vocabulary includes 1,000 time features and two speaker features to encode when and who produced each utterance.
- The controller computes attention over memory, returns a weighted memory vector, and can reread the memory for a fixed number of hops.Performance improves empirically with up to 3 or 4 hops.
- The final prediction selects among candidate bot utterances and API calls, and the model is trained with cross-entropy loss using stochastic gradient descent.
B EXAMPLES OF PREDICTIONS OF A MEMORY NETWORK
Prediction examples show Memory Networks using attention over dialog memories across several tasks and on Concierge dialogs.
- Examples report predictions and memory attention for Tasks 1–4 using a three-hop Memory Network, and for Concierge using a two-hop model.
C HYPERPARAMETERS
The paper selects supervised-embedding and Memory Network hyperparameters separately for each task using validation performance.
- The best supervised-embedding and Memory Network models for each task are selected using the corresponding validation sets.
D ADDITIONAL RESULTS
Additional results compare model variants and illustrate task-specific behavior, including attention patterns, hyperparameters, anonymized Concierge predictions, and out-of-vocabulary performance.
- On some tasks, supervised embeddings work better with only the latest user utterance, while bigrams generally hurt performance.Match type features improve out-of-vocabulary Tasks 1 and 5 but remain behind Memory Networks with match type features.
- For Task 1, attention concentrates on four memories containing information needed to issue the API call, and more hops strengthen this signal.
- For Task 2, attention correctly focuses on the original API call and the utterance providing the update.
- For Task 3, the model recognizes that it should display options but fails to attend reliably to ratings, so the task remains unsolved overall.
- For Task 4, match type features solve confusion between phone numbers and addresses by emphasizing entities appearing in the history.
- On Concierge dialogs, the model produces semantically equivalent top-five answers using anonymized tokens for people, organizations, numbers, and dates.
- Table 10 reports standard and out-of-vocabulary results across tasks, with per-response accuracy and per-dialog accuracy shown for each method.