Source-linked AI summary
Large-scale Simple Question Answering with Memory Networks
Antoine Bordes, Nicolas Usunier, Sumit Chopra, Jason Weston
TL;DR
The paper studies how to improve large-scale simple question answering when training data covers only a limited range of possible questions. It introduces a large dataset and evaluates Memory Networks with multitask and transfer learning, finding strong benchmark performance and cross-dataset benefits. The authors also demonstrate transfer from Freebase to Reverb without retraining.
Problem
Existing QA benchmarks cover limited question distributions, leaving performance outside their templates and transfer across datasets uncertain.
Method
The paper introduces SimpleQuestions and trains embedding-based Memory Networks using multiple QA data sources, including Freebase and Reverb transfer.
Results
The model achieves state-of-the-art on WebQuestions, improves on both datasets with joint training, and answers using Reverb facts without retraining.
Takeaways & Limitations
Properly trained Memory Networks can handle natural language and memories containing millions of entries for large-scale simple QA.
Takeaways & Limitations
WebQuestions provides answer strings but not the supporting facts, so training requires weak or distant supervision rather than direct fact supervision.
Abstract
from arXiv · showhide
Training large-scale question answering systems is complicated because training sources usually cover a small portion of the range of possible questions. This paper studies the impact of multitask and transfer learning for simple question answering; a setting for which the reasoning required to answer is quite easy, as long as one can retrieve the correct evidence given a question, which can be difficult in large-scale conditions. To this end, we introduce a new dataset of 100k questions that we use in conjunction with existing benchmarks. We conduct our study within the framework of Memory Networks (Weston et al., 2015) because this perspective allows us to eventually scale up to more complex reasoning, and show that Memory Networks can be successfully trained to achieve excellent performance.
1 Introduction
The paper addresses simple question answering over knowledge bases, where retrieving one supporting fact remains difficult at large scale. It introduces SimpleQuestions and evaluates Memory Networks with multitask and transfer learning across QA datasets.
- Motivation: Simple Question Answering remains unsolved despite requiring only one knowledge-base fact, because retrieving that fact from natural-language questions is difficult.The paper distinguishes simple reasoning from challenging large-scale retrieval.
- Motivation: Existing benchmarks are small and narrowly varied, leaving cross-dataset transfer and performance beyond benchmark-specific question templates unknown.They mostly cover frequent facts and limited question types, syntactic forms, and lexical variations.
- Contributions: SimpleQuestions contains more than 100k human-written questions paired with Freebase facts, whereas WebQuestions contains fewer than 6k automatically generated questions.The dataset was introduced to study broader coverage and joint training across data sources.
- Contributions: The authors develop an embedding-based question-answering system within the Memory Networks framework, embedding natural language and knowledge-base facts in one vector space.The simple QA setting corresponds to retrieving the relevant fact from memory.
- Results: The model achieves excellent WebQuestions results, improves on both datasets when trained jointly, and transfers from Freebase to Reverb without retraining.Reverb facts are added to memory for answering, and the model outperforms some systems designed for Reverb.
2 Simple Question Answering
Simple QA asks for objects linked to a subject by a relationship, so one fact can answer the question, but retrieval remains difficult over large knowledge bases. The paper uses Freebase and Reverb and introduces the large SimpleQuestions dataset to broaden evaluation and study transfer.
- Simple QA: Simple QA rephrases a natural-language question as a query (subject, relationship, ?), returning all linked objects from a knowledge base.Fetching one supporting fact is sufficient to answer the question.
- Simple QA: Although the reasoning process uses a single fact, retrieving that fact from millions of alternatives can be challenging.The paper notes that knowledge-base organization can broaden the range of questions answerable with one fact.
- Knowledge Bases: Freebase supplies typed entities and relationships, with FB2M containing about 2M entities and 5k relationships and FB5M about 5M entities and more than 7.5k relationships.The paper uses Freebase as its primary source of facts and answers.
- Knowledge Bases: Reverb provides a contrasting secondary knowledge base with automatically extracted, highly unstructured facts, open relationship vocabulary, and lower fact reliability.Its extraction contains 2M entities and 600k relationships, and it supports a transfer-learning experiment without Reverb training.
- The SimpleQuestions dataset: SimpleQuestions contains 108,442 human-written questions, each paired with a corresponding FB2M fact that provides and explains the answer.The data is split into 70% training, 10% validation, and 20% test sets.
- The SimpleQuestions dataset: The dataset construction filters Freebase facts to remove undefined relationships and subject-relationship pairs with more than ten objects before human question generation.The threshold removes trivial, uninformative questions.
3 Memory Networks for Simple QA
The system uses a Memory Network to retrieve a supporting Freebase fact for each question, after preprocessing and embedding questions, facts, and external Reverb facts. Candidate generation narrows the search, scoring selects the most similar fact, and the response returns its objects.
- Workflow: A three-stage workflow stores Freebase, trains the network, and then connects Reverb facts without retraining.The final answering process runs the Input, Output, and Response modules in sequence.
- Input module: Freebase facts are grouped by subject and relationship so list questions can return all linked objects.Grouping reduces FB2M from 14M to 11M facts and FB5M from 22M to 12M.
- Input module: Removing mediator nodes creates direct links and raises WebQuestions questions answerable with one fact from around 65% to 86%.This preprocessing reduces the need to search beyond the subject’s immediate neighborhood.
- Representations: Questions use bag-of-ngrams vectors, while Freebase and Reverb facts use bag-of-symbol or bag-of-words representations.The representations enable common embedding-based scoring across question and fact inputs.
- Output module: The Output module links question n-grams to entity aliases, scores candidate facts by cosine similarity, and selects one supporting fact.It avoids scoring every stored fact by first generating a small candidate set.
- Response and generalization: The Response module returns the objects of the selected supporting fact as the answer.Reverb facts are connected to the Freebase memory through entity linking or string-based representations.
4 Training
Training alternates optimization across QA, automatically generated, and paraphrase data so shared embeddings learn to rank supporting facts and paraphrases above negatives. WebQuestions receives distant supervision because it provides answer strings rather than supporting facts.
- Multitask data: The scoring function is trained multitask on four data sources: SimpleQuestions, WebQuestions, synthetic KB questions, and paraphrases.The synthetic data supplies entity coverage, while the paraphrase set contains 15M clusters.
- Ranking objective: Ranking training makes supporting facts more similar to questions than non-supporting facts, and paraphrases more similar than unrelated questions.The embedding matrices are optimized by alternating stochastic-gradient steps across datasets.
- Optimization: Paraphrase samples receive probability 0.2 during training, while QA samples are drawn uniformly from three QA datasets.Training uses WARP, Adagrad, and HogWild!; it takes 2–3 hours on 20 threads.
- Distant supervision: WebQuestions uses answer strings rather than supporting facts, so candidate facts are selected by matching object aliases to the provided answers.Questions without matching answer strings among initial candidates are discarded from training.
- Limitation: Future work should examine weakly supervised MemNN training that does not require supporting-fact supervision.This is identified as a direction for addressing the supervision requirement.
- Negative examples: Candidates-as-negatives samples a non-supporting fact from the question’s candidate set and performs slightly better than corrupting a fact.The alternative corruption policy changes a subject, relationship, or object, sometimes changing multiple elements.
5 Related Work
Related QA systems use search, semantic parsing, or embeddings, trading off scalability, interpretability, and representation choices. The paper positions its approach among embedding-based Memory Network methods while using Freebase and richer representations.
- Search-based approaches: Search-based QA extracts answers from top search results and has been adapted to knowledge-base QA with competitive performance.These systems send question keywords to a search engine before extracting an answer.
- Semantic parsing: Semantic parsing converts questions into executable knowledge-base queries, providing deep interpretation but making large-scale training difficult.Some methods reduce supervision needs by using rules to transform interpretations into KB queries.
- Embedding methods: Embedding-based approaches can be viewed as simple Memory Networks, but prior systems differ in knowledge base, question representation, and training data.The cited prior methods use Reverb or entity-token and Wikipedia-based representations rather than the paper’s setup.
6 Experiments
The experiments evaluate Memory Networks across WebQuestions, SimpleQuestions, and Reverb, including alternative configurations, multiple training sources, and transfer without retraining. Results show strong WebQuestions performance, improved cross-dataset learning, and successful use of newly added Reverb facts, while Simple QA remains unsolved.
- Comparative results: F1-scores of 41.9% and 42.2% are achieved by the two best ensembles on WebQuestions, exceeding the 41.3% score of the best published competing approach.A single run of the model reaches 41.2% F1-score.
- Comparative results: 62−63% accuracy is achieved on SimpleQuestions, although the supporting fact appears among candidates for about 86% of questions.The result indicates effective candidate re-ranking but also that simple QA remains unsolved.
- Transfer learning on Reverb: 67% accuracy is obtained on Reverb without retraining, or 68% with five models, compared with 54% for the original paper and 73% state of the art.The experiment adds all Reverb facts to memory and evaluates answer re-ranking on the companion QA set.
- Importance of data sources: Training on both WebQuestions and SimpleQuestions improves performance on both datasets and captures their question patterns without negative interaction.Training on WebQuestions alone yields 46.6% accuracy on SimpleQuestions, illustrating limited cross-dataset coverage from single-source training.
- Importance of data sources: Paraphrases have a dramatic impact on Reverb performance but provide little benefit on WebQuestions and SimpleQuestions, whose questions follow simpler, well-formed patterns.The authors connect this difference to Reverb’s greater syntactic and lexical variability and identify paraphrases as important for avoiding overfitting to training patterns.
7 Conclusion
The paper shows that properly trained Memory Networks can handle natural language with memories containing millions of entries and reach state-of-the-art WebQuestions performance. It also introduces SimpleQuestions as a substantially larger resource intended to support further QA research.
- Conclusion: Properly trained Memory Networks handle natural language and very large memories containing millions of entries, reaching state-of-the-art performance on WebQuestions.The conclusion presents this as the paper’s main result for large-scale simple QA.
- Conclusion: SimpleQuestions contains 100k examples, making it one order of magnitude larger than WebQuestions and a resource for further QA research.The authors expect the dataset to foster research on simple QA and broader QA settings.