Source-linked AI summary
Question Answering on Freebase via Relation Extraction and Textual Evidence
Kun Xu, Siva Reddy, Yansong Feng, Songfang Huang, Dongyan Zhao
TL;DR
Knowledge-based question answering needs methods that are both data-efficient and capable of handling compositional constraints. This paper retrieves candidates from Freebase with neural relation extraction and validates them using Wikipedia evidence, achieving 53.3% F_1 on WebQuestions. The authors report that this combined approach reaches state-of-the-art performance.
Problem
Semantic parsing requires extensive annotated compositional data, while relation extraction is less expressive for questions involving multiple constraints.
Method
The method jointly performs entity linking and neural relation extraction on Freebase, then uses Wikipedia inference to validate and prune candidate answers.
Results
53.3% F_1 on WebQuestions improves the default pipelined model from 44.1% to 53.3% and achieves a new state-of-the-art result.
Takeaways & Limitations
Unstructured inference helps mitigate representational issues in structured inference, while the combined model achieves state-of-the-art WebQuestions results.
Takeaways & Limitations
The method still fails on compositional multi-hop questions, and its evidence assumption may work only for frequently typed queries or popular domains.
Abstract
from arXiv · showhide
Existing knowledge-based question answering systems often rely on small annotated training data. While shallow methods like relation extraction are robust to data scarcity, they are less expressive than the deep meaning representation methods like semantic parsing, thereby failing at answering questions involving multiple constraints. Here we alleviate this problem by empowering a relation extraction method with additional evidence from Wikipedia. We first present a neural network based relation extractor to retrieve the candidate answers from Freebase, and then infer over Wikipedia to validate these answers. Experiments on the WebQuestions question answering dataset show that our method achieves an F_1 of 53.3%, a substantial improvement over the state-of-the-art.
1 Introduction
Knowledge-base question answering balances expressive semantic parsing against data-efficient relation extraction, but compositional questions remain difficult. The paper combines Freebase retrieval with Wikipedia evidence to validate candidate answers.
- Motivation: Semantic parsing supports sophisticated meaning representations but requires many annotated compositional examples and can mismatch knowledge-base structure.
- Motivation: Relation extraction and distributed-representation methods are easier to train but struggle with questions requiring multiple entities, relations, or latent constraints.
- Textual Evidence: Wikipedia sentences can provide external evidence that filters incorrect Freebase candidates and supplies additional constraints, such as an answer’s gender.
- Contribution: The proposed system jointly performs entity linking and relation extraction on Freebase, then validates candidate answers against Wikipedia.
2 Our Method
The method first generates candidate answers through Freebase inference and then refines them using evidence from the topic entity’s Wikipedia page. It also decomposes multi-relational questions and intersects sub-question answers.
- Freebase Inference: Freebase inference links question entities, predicts possible relations, and jointly selects an entity-relation configuration producing candidate answers.
- Answer Refinement: Wikipedia answer refinement uses the topic entity’s page to filter wrong candidates and select supported answers.
- Multi-relational Questions: Multi-relational questions are decomposed into sub-questions, whose answers are independently obtained from Freebase and Wikipedia.
- Multi-relational Questions: The final answer for a multi-relational question is the intersection of the answer sets from all sub-questions.
3 Inference on Freebase
Freebase inference formulates question answering as entity linking and relation extraction, then jointly ranks entity-relation pairs using neural, lexical, and answer-level clues.
- Formulation: Each sub-question is mapped to a single Freebase triple (e, r, ?), making entity linking and relation extraction the two core inference tasks.
- Entity Linking: The entity linker retrieves and ranks candidate Freebase entities for each detected named-entity mention.
- Relation Extraction: The MCCNN relation extractor combines syntactic dependency-path and sentential channels to predict relation confidence scores.
- Objective Function and Learning: The relation model is trained with cross-entropy against a one-hot target relation distribution, using back-propagation and AdaGrad.
- Joint Inference: Joint inference ranks entity-relation pairs globally, exploiting entity, relation, and answer clues rather than optimizing linking and extraction independently.
4 Inference on Wikipedia
Wikipedia inference refines Freebase candidates by classifying evidence sentences associated with the topic entity. Positive evidence supports a candidate, while absent evidence triggers fallback to the knowledge-base result.
- Answer Refinement: The best Freebase entity-relation pair supplies candidate answers for subsequent Wikipedia validation.
- Finding Evidence: The system maps the topic entity to Wikipedia, processes pages with entity linking and coreference resolution, and retrieves sentences containing candidate answers.
- Refinement Model: The refinement model treats sentences containing correct answers as positive examples and sentences containing incorrect answers as negative examples.
- Decision Rule: A candidate is accepted when at least one positive evidence sentence supports it; if no evidence is found, the system falls back to the KB-based result.
- Lexical Features: Lexical features are token-pair occurrences between question and evidence-sentence words, allowing patterns such as (first, drafted) to receive learned weights.
5 Experiments
This section introduces the experimental setup, reports the main results, and provides detailed analysis of the system.
- The section covers the experimental setup, main results, and detailed system analysis.
- The experiments evaluate the proposed system through a defined setup and reported results.
- The section includes detailed analysis alongside the experimental results.
5.1 Training and Evaluation Data
The experiments use WebQuestions, with question-answer annotations and predefined training, development, and test splits. Training relations are obtained through surrogate gold relations selected by answer overlap with the gold answers.
- WebQuestions contains 5,810 questions crawled through Google Suggest and annotated with answers on Amazon Mechanical Turk.
- The dataset is split into 3,778 training questions and 2,032 test questions, with 80%/20% of training used for development.
- Surrogate gold relations are selected by querying 1-hop and 2-hop Freebase candidates and minimizing F1 loss against gold answers.
- The training set provides 461 relations for MCCNN training and testing-time prediction.
5.2 Experimental Settings
The experimental settings specify dependency-based question decomposition, word-embedding initialization, model hyperparameters, and the Freebase version used.
- Six dependency tree patterns based on Bao et al. decompose questions into subquestions.
- Word embeddings are initialized with Turian et al. representations using 50 dimensions.
- The MCCNN window size is 3, with hidden-layer sizes of 200 and 100 for both channels.
- The model hyperparameters are tuned on the development set.
- The system uses the Freebase version of Berant et al., containing 4M entities and 5,323 re.
5.3 Results and Discussion
On WebQuestions, joint entity linking and relation extraction improves Freebase-only QA, while Wikipedia-based answer refinement further raises performance and addresses some representational weaknesses. Error analysis shows remaining failures arise mainly from relation prediction, entity linking, and insufficient evidence or compositional coverage.
- Test-set results: 53.3% F1 is achieved by the main Structured + Joint + Unstructured model, improving the default pipelined model from 44.1% by 9.2%.It combines joint inference on Freebase with Wikipedia-based validation and achieves a new state-of-the-art result.
- Impact of Joint EL & RE: 3% performance gain results from joint entity linking and relation extraction, increasing F1 from 44.1 to 47.1.Joint inference outperforms the default pipelined approach.
- Impact of Joint EL & RE: Joint inference raises entity-linking accuracy from 79.8% to 83.2% and relation-prediction accuracy from 45.9% to 55.3% on development data.The respective improvements are 3.4% and 9.4%.
- Impact of the Syntactic and the Sentential Channels: Using both MCCNN channels improves QA performance beyond either single channel, with sentential features more important than syntactic features.The authors attribute this to the short, noisy nature of WebQuestions questions and parsing limitations.
- Impact of the Inference on Unstructured Data: 6.2% additional improvement comes from unstructured inference after Structured + Joint, increasing performance from 47.1% to 53.3%.Wikipedia evidence mainly helps aggregation questions and sub-lexical compositionality, including distinctions that Freebase relations do not encode.
- Error analysis: About 50% of errors come from incorrect relation predictions, around 15% from entity linking, and remaining errors from insufficient Wikipedia evidence or misclassification.Relation errors reflect insufficient context and imbalanced relation frequencies.
6 Related Work
The paper situates its approach between semantic parsing and relation extraction, combining structured and unstructured resources for question answering.
- Structured question answering research has shifted from small toy domains toward large-scale and noisy knowledge bases such as Freebase and DBpedia.
- This work differs from earlier hybrid approaches by using unstructured data at test time to validate structured candidates.
- Unlike related sentence-level extractors, the model predicts multiple relations per question and uses an MCCNN architecture for relation prediction.
7 Conclusion and Future Work
The paper combines structured and unstructured inference for natural-language question answering and reports state-of-the-art WebQuestions results, while noting coverage limitations from Freebase.
- The method infers over structured and unstructured data, using joint entity linking and relation extraction followed by unstructured inference.
- Unstructured inference helps mitigate representational issues in structured inference, including knowledge gaps between the two resources.
- The main model achieves state-of-the-art results on the WebQuestions dataset.
- Because structured inference is performed first, the method remains limited by Freebase coverage.
Appendix
The appendix describes syntax-based patterns for decomposing questions into sub-questions, covering both simple questions and clause-based complex questions.
- The first four syntax-based patterns extract sub-questions from simple questions.
- The latter two patterns handle complex questions involving clauses.