Source-linked AI summary
Open Question Answering with Weakly Supervised Embedding Models
Antoine Bordes, Jason Weston, Nicolas Usunier
TL;DR
Open question answering systems often require substantial labeled data or hand-crafted lexicons and grammars, limiting schema-independent use across knowledge bases. This paper learns vector embeddings for questions and answers from weak supervision, combining automatically generated and collaboratively marked data with fine-tuning, and significantly outperforms paralex on simple factual questions.
Problem
Open question answering remains challenging because semantic parsers require substantial human intervention through labeled data, hand-crafted lexicons, grammars, and knowledge-base schemas.
Method
The model maps questions and knowledge-base triples into a shared embedding space, training with automatically generated question-triple pairs, paraphrase data, stochastic gradient descent, and fine-tuning.
Results
The embedding framework significantly outperforms paralex for answering simple factual questions, with most improvement attributed to better recall.
Takeaways & Limitations
Weak and indirect supervision can train meaningful embedding models for open question answering despite imperfect labeled data.
Takeaways & Limitations
The model satisfactorily answers only simple factual questions and does not account for word ordering or the semantics of more complex questions.
Abstract
from arXiv · showhide
Building computers able to answer questions on any subject is a long standing goal of artificial intelligence. Promising progress has recently been achieved by methods that learn to map questions to logical forms or database queries. Such approaches can be effective but at the cost of either large amounts of human-labeled data or by defining lexicons and grammars tailored by practitioners. In this paper, we instead take the radical approach of learning to map questions to vectorial feature representations. By mapping answers into the same space one can query any knowledge base independent of its schema, without requiring any grammar or lexicon. Our method is trained with a new optimization procedure combining stochastic gradient descent followed by a fine-tuning step using the weak supervision provided by blending automatically and collaboratively generated resources. We empirically demonstrate that our model can capture meaningful signals from its noisy supervision leading to major improvements over paralex, the only existing method able to be trained on similar weakly labeled data.
1 Introduction
Open-domain question answering seeks systems that answer questions across domains using large knowledge bases, but natural-language interpretation and scale remain difficult. Existing semantic-parsing methods can query such bases yet depend on hand-crafted lexicons, grammars, and schemas, motivating vectorial representations trained with weak supervision.
- 1 Introduction: Open-domain question answering aims to build systems able to answer questions from any domain using large-scale knowledge bases.These knowledge bases store general-purpose information as triples connecting entities through relationships.
- 1 Introduction: Semantic parsers convert questions into logical forms or database queries, but effective deployment requires hand-crafted lexicons, grammars, and knowledge-base schemas.This human intervention may not scale conveniently to new domains.
- 1 Introduction: The paper maps questions to uninterpretable vectorial representations that require no predefined grammars or lexicons and work independently of a knowledge base’s schema.Answers are represented in the same space so the knowledge base can be queried directly.
- 1 Introduction: The target task focuses on simple factual questions whose answers are represented by one or more knowledge-base triples.Its central difficulties include lexical variability, multiple answers, and the absence of supervised training data.
- 1 Introduction: The model learns meaningful question and triple representations from automatically generated and collaboratively labeled resources, improving substantially over paralex on weakly labeled evaluation data.The training resources combine generated question–triple pairs with collaboratively marked question paraphrases.
2 Related Work
Related work spans web-search systems, knowledge-base question answering, semantic parsing, and embedding models. The paper builds on weakly supervised open question answering and extends embedding-based links between language and knowledge bases to this task.
- 2 Related Work: Early large-scale question-answering systems transformed questions into web-search queries and extracted answers from returned pages or snippets.These systems required substantial engineering to craft queries and parse search results.
- 2 Related Work: Large knowledge bases reframed open question answering as natural-language querying, but language variability and knowledge-base scale make purely supervised learning data-intensive.Hand-written templates were not robust to evolving knowledge bases, motivating distant or indirect supervision.
- 2 Related Work: Semantic-parsing systems reduce labeling needs but still require considerable manual design of examples, lexicons, grammars, and knowledge-base resources.Paralex instead targets limited semantics with little human annotation at very large open-domain scale.
- 2 Related Work: This work applies embedding models to open question answering under weak supervision, extending prior language–knowledge-base connections beyond word-sense disambiguation and information extraction.The authors state that this application had not previously been attempted to their knowledge.
3 Open-domain Question Answering
The task ranks knowledge-base triples as answers to natural-language questions, using a scoring function rather than an intermediate logical form. Training combines automatically generated question–triple pairs from ReVerb with collaboratively labeled paraphrases, despite substantial noise in the resulting data.
- 3.1 Task Definition: The task represents each answer with a knowledge-base triple and allows multiple triples to interpret a single question.Examples include triples encoding a dodo’s environment, Hannukah’s symbols, or a laser’s use.
- 3.1 Task Definition: The model learns a scoring function S(·) for question–triple pairs and ranks candidate triples directly, avoiding an intermediate structured logical representation.Multiple answers are returned as a ranked list rather than only the top prediction.
- 3.2 Training Data: ReVerb supplies more than 14M automatically extracted triples involving more than 2M entities and 600k relationships.Its broad coverage and limited human intervention suit the weakly supervised setting, although the resource is noisy and ambiguous.
- 3.2 Training Data: Approximately 16 × 14M question–triple pairs are generated by applying 16 seed-question patterns to randomly selected ReVerb triples.The generated questions provide positive training pairs but form an imperfect, noisy weak signal because their syntax and entity typing can be invalid.
- 3.2 Training Data: ReVerb’s untyped entity names and stripped suffixes cause incoherent generated questions, while richer curated knowledge bases could improve quality at the cost of more human intervention.The paper retains ReVerb to preserve its low-intervention training motivation.
- 3.2 Training Data: Collaboratively labeled WikiAnswers paraphrase pairs supplement generated examples to connect knowledge-base triples with more varied natural language.The harvested resource contains 18M pairs and 2.4M distinct questions, but only an estimated 55% of pairs are actual paraphrases.
4 Embedding-based Model
The model maps questions and knowledge-base triples into a shared embedding space, scores their similarity, and predicts answers by ranking triples. It learns from positive pairs, corrupted negatives, and paraphrase pairs, then fine-tunes the similarity function to improve ranking.
- Architecture: Questions and KB triples are projected into a shared embedding space, where their dot product scores compatibility.Questions use bag-of-words representations, while triples combine entity and relationship embeddings; entities use distinct embeddings for left- and right-hand positions.
- Architecture: At test time, the model predicts an answer by selecting the highest-scoring triple from the KB.This directly queries the KB without an intermediate logical representation.
- Training: Training uses a ranking loss that separates each positive question-triple pair from corrupted triples by a margin of 0.1.Corrupted triples are created by replacing triple components with corresponding elements from a random KB triple, and embedding-vector norms are constrained.
- Training: The model jointly trains on pseudolabeled question-triple data and paraphrase pairs, sharing the word-embedding matrix across both objectives.Paraphrase negatives are formed by replacing one question with another randomly selected question.
- Training: The large problem scale motivates a simple SGD-based training procedure over approximately 4.3M embeddings and 250M combined examples.The embedding dimension is 64, and Adagrad adapts the SGD learning rate.
- Fine-tuning: A learned matrix M fine-tunes the similarity between question and triple embeddings, producing consistent performance improvements through slight ranking changes.M is learned with fixed embedding matrices using a convex optimization problem and does not define a dot product because it is not symmetric.
5 Experiments
Experiments evaluate the embedding model through reranking candidate triples, full ranking over ReVerb, learned embeddings, and transfer to WebQuestions. Results show gains from paraphrase multitasking, fine-tuning, and candidate filtering, while performance remains constrained by noisy data and task mismatch.
- Evaluation Protocols: The evaluation measures reranking quality on WikiAnswers+ReVerb using precision, recall, F1-score, and MAP over candidate triples.The test set contains 691 questions with 48k hand-labeled candidate triples.
- Reranking: F1 rises from 0.60 to 0.68 when the model is trained multitask with paraphrase data.The authors attribute this improvement to richer connections between KB constituents and words.
- Full Ranking: String matching reduces the average ranked set from 14M to about 10k triples and improves both precision and recall while reducing evaluation time.Candidate strings are derived from noun phrases and proper nouns before model scoring.
- Evaluation on WebQuestions: On WebQuestions, the model transfers without labeled retraining, but performance is not great because many questions require multiple triples.The evaluation maps 1,538 of 2,034 questions to ReVerb entities and may miss answers represented under alternate names.
6 Conclusion
The paper presents an embedding-based framework for open question answering that learns from imperfect and indirect supervision, outperforming previous work on simple factual questions. Its scope remains limited to simple semantics, leaving complex question understanding unresolved.
- The framework uses embeddings to learn open question answering from imperfect labeled data and indirect supervision.
- The approach significantly outperforms previous work when answering simple factual questions.
- The paper introduces a fine-tuning method for embedding models when their optimization problem cannot be completely solved.
- The model satisfactorily handles only simple factual questions because its supervision signal is very limited.
- The model does not account for word ordering and requires further work to represent more complex question semantics.