Source-linked AI summary
Improved Neural Relation Detection for Knowledge Base Question Answering
Mo Yu, Wenpeng Yin, Kazi Saidul Hasan, Cicero dos Santos, Bing Xiang, Bowen Zhou
TL;DR
KBQA relation detection must handle large relation vocabularies, unseen relations, and multi-relation chains. This paper addresses these challenges with a residual hierarchical BiLSTM matcher and a simple entity-linking-integrated KBQA system, achieving state-of-the-art outcomes across single- and multi-relation benchmarks.
Problem
KBQA relation detection must handle thousands of relation types, unseen test relations, and relation chains, unlike many general relation-extraction settings.
Method
The paper hierarchically matches questions with relation-level and word-level representations using deep residual BiLSTMs, integrating the detector with entity linking in a simple KBQA system.
Results
The improved detector outperforms previous methods on KB relation detection and enables state-of-the-art KBQA results on SimpleQuestions and WebQSP.
Takeaways & Limitations
Improved KB relation detection can benefit both relation-detection performance and the KBQA end task.
Takeaways & Limitations
Attention-based hierarchical matching did not improve results over HR-BiLSTM, and further attention mechanisms are left for future work.
Abstract
from arXiv · showhide
Relation detection is a core component for many NLP applications including Knowledge Base Question Answering (KBQA). In this paper, we propose a hierarchical recurrent neural network enhanced by residual learning that detects KB relations given an input question. Our method uses deep residual bidirectional LSTMs to compare questions and relation names via different hierarchies of abstraction. Additionally, we propose a simple KBQA system that integrates entity linking and our proposed relation detector to enable one enhance another. Experimental results evidence that our approach achieves not only outstanding relation detection performance, but more importantly, it helps our KBQA system to achieve state-of-the-art accuracy for both single-relation (SimpleQuestions) and multi-relation (WebQSP) QA benchmarks.
1 Introduction
KBQA parses questions by linking entities and detecting the relations they express, but KB-specific detection must handle many, unseen, and chained relations. The paper improves relation detection with hierarchical matching and residual learning, then integrates it into a simple KBQA system.
- Motivation: KBQA systems generate executable knowledge-base queries from input questions to retrieve answers from knowledge-base tuples.The illustrated pipeline includes entity linking and relation detection before querying the knowledge base.
- Motivation: KB relation detection differs from general relation detection because knowledge bases contain thousands of relation types, unseen test relations, and multi-relation chains.Freebase2M contains more than 6,000 relation types, and 14% of golden SimpleQuestions test relations were absent from golden training tuples.
- Approach: The proposed detector matches questions with relation names at multiple abstraction levels using deep bidirectional LSTMs and residual learning.It combines relation-level and word-level representations while residual sequence matching facilitates deeper, more abstract question representations.
- Approach: The simple KBQA system uses entity linking and two-step relation detection, including relation-based re-ranking of entity candidates before querying the knowledge base.The detector scores relations connected to candidate entities and helps resolve entity ambiguities through re-ranking.
- Results: The improved relation detector enables state-of-the-art results on both single-relation and multi-relation KBQA tasks.The contribution is evaluated on SimpleQuestions and WebQSP.
2 Related Work
Related work spans general relation extraction and KBQA relation detection, but the two settings differ in relation-vocabulary size, unseen-relation demands, and available entity information.
- Relation Extraction: General relation extraction classifies relations between two target entities, commonly using a small predefined relation set.Recent methods include CNNs, LSTMs, and attention models.
- Relation Extraction: Common general relation-extraction benchmarks contain far fewer relation types than KBQA settings with thousands of relations.ACE2005 has 11/32 coarse/fine-grained relations, SemEval2010 Task8 has 19, and TAC-KBP2015 has 74.
- Relation Detection in KBQA Systems: KBQA relation detection has progressed from feature-rich methods to deep networks and attention models, with varying support for open-domain relation sets.Dataset requirements differ: WebQuestions often permits a close-domain assumption, whereas SimpleQuestions and ParaLex require stronger support for large and unseen relation sets.
- Relation Detection in KBQA Systems: Unlike general relation extraction, KBQA usually lacks both argument entities and reliable entity-type information for relation-detection features.Questions often contain only a topic entity, and a knowledge-base entity may have multiple types.
3 Background: Different Granularity in KB Relations
KB relations can be represented either as whole relation tokens or as word sequences, capturing complementary global and local information. The paper therefore uses hierarchical matching across both granularities.
- Relation Representations: Prior KBQA work formulates relation detection as sequence matching, making relation representation a central modeling challenge.Questions are naturally word sequences, whereas relations require a chosen sequence representation.
- Relation Representations: Treating each relation name as one token preserves global information but generalizes poorly when relation-specific training data are sparse.Unseen relation names receive poorly learned embeddings and may have low coverage.
- Relation Representations: Representing relations as word sequences improves generalization but can lose global relation-name information and confuse semantically similar candidates.Word-level matching may rank “plays produced” above the target “starring roles.”
- Complementary Granularities: Word-level representations emphasize local words and short phrases, while relation-level representations capture longer phrases and skip-grams but suffer from data sparsity.The two granularities provide complementary abstractions for question-relation matching.
- Complementary Granularities: The proposed hierarchical matcher scores each candidate relation against both word-level and relation-level representations.Table 1 illustrates relation-name and word tokens, with italicized question phrases serving as evidence for their matches.
4 Improved KB Relation Detection
The paper represents relations at word and relation-name granularities, questions at multiple abstraction levels, and matches them hierarchically using residual BiLSTMs.
- 4.1 Relation Representations from Different Granularity: Relations are represented with both word-level tokens and relation-level names, supporting matching at different granularities.Word sequences include components such as “episode” and “written,” while relation-level representations include names such as “episode written” or chained relations.
- 4.1 Relation Representations from Different Granularity: Two shared-parameter BiLSTMs encode relation sequences, initialize relation-level processing from word-sequence states, and max-pool the resulting vectors.Initializing relation sequence LSTMs with word-sequence final states provides a back-off for unseen relations.
- 4.2 Different Abstractions of Questions Representations: Different relation components can match question phrases of different lengths, motivating question representations that summarize information at multiple abstraction levels.Relation names may match longer phrases, while relation words may match shorter or differently sized phrases.
- 4.2 Different Abstractions of Questions Representations: Deep BiLSTMs produce first- and second-layer question representations, with the second layer intended to capture more general and abstract information.Either question layer may match either relation representation level rather than corresponding to only one fixed granularity.
- 4.3 Hierarchical Matching between Relation and Question: The HR-BiLSTM uses shortcut connections to preserve hierarchical matching; without them, the model compares only second-layer question representations with the relation.The model is trained with a ranking loss that maximizes the margin between gold and candidate relations.
- 4.3 Hierarchical Matching between Relation and Question: Naive hierarchical matching can suffer training difficulty because one layer may dominate while the other receives near-zero weight, and deeper architectures are harder to train.The paper reports that this approach does not significantly improve performance and may converge to a higher training loss than a single-layer baseline.
- 4.3 Hierarchical Matching between Relation and Question: Residual shortcut connections make the question-layer representations complementary by fitting the second layer to the first layer’s matching residues.The final question representation is max-pooled, and the relation-question matching score is computed with cosine similarity.
5 KBQA Enhanced by Relation Detection
The KBQA pipeline uses relation detection twice: first to re-rank linked entity candidates, then to detect relations for the reduced candidate set and generate a query. This interaction helps resolve entity ambiguity while combining entity-linking and relation scores for final selection.
- Pipeline: The pipeline starts with top-K entities from an existing entity linker and generates KB queries through entity re-ranking, relation detection, and query generation.An optional constraint-detection step can add neighboring entities and relations to the query.
- 5.1 Entity Re-Ranking: Detected relations help disambiguate entities because different candidates typically connect to different relations in the knowledge base.The paper gives the example that relations such as “episodes written,” “author of” and “profession” favor the TV writer over the baseball player with the same name.
- Pipeline: The authors describe the two-step process as mutually beneficial: relation detection improves entity re-ranking, and re-ranked entities enable more accurate relation detection.The stated result is that the KBQA end task benefits from this process.
- 5.1 Entity Re-Ranking: Relation detection scores KB relations connected to at least one initial entity candidate, and these scores re-rank the entity list.The system retains a shorter top-K′ candidate list after re-ranking.
- 5.2 Relation Detection: The second relation-detection step replaces the candidate entity mention with <e> so the model can use the relative position of each question word.It scores relations associated with each candidate entity using the reformatted question.
- 5.3 Query Generation: Query generation selects the final entity-relation pair by combining the entity re-ranking score with the relation score, using β to balance them.The system outputs the pair (ê, r̂), while an optional constraint step can extend the query with high-scoring neighboring entities and relations.
6 Experiments
Experiments evaluate relation detection and KBQA end-task performance on SimpleQuestions and WebQSP. The hierarchical residual BiLSTM improves relation detection and raises KBQA accuracy, while ablations clarify the roles of relation representations, depth, and residual connections.
- Task Introduction & Settings: Experiments use SimpleQuestions for single-relation QA and WebQSP for multi-relation QA, evaluating relation detection independently and within KBQA.Both datasets provide gold semantic parses; WebQSP evaluation includes relation chains of length at most 2.
- Relation Detection Results: The HR-BiLSTM outperformed the best relation-detection baselines on both tasks by 2-3%.The improvements were statistically significant compared with the best baseline BiLSTM with words on both SimpleQuestions and WebQSP.
- Relation Detection Results: 91.2% to 88.9% shows that using only relation names weakens the baseline BiLSTM substantially on SimpleQuestions.The smaller WebQSP decrease suggests unseen relations have a larger impact on SimpleQuestions.
- Relation Detection Results: 93.3% versus 91.2%/88.8% shows that hierarchical matching of questions with relation names and relation words improves both datasets, especially SimpleQuestions.The ablation compares the proposed hierarchical representation against baselines using weaker relation representations.
- Relation Detection Results: WebQSP benefits more from residual and deeper architecture than SimpleQuestions, where removing deep layers causes only a small performance drop.Replacing BiLSTM with CNN also causes a large WebQSP drop but a much smaller SimpleQuestions gap, possibly because chains require longer dependencies.
- Relation Detection Results: The two-layer BiLSTM without shortcut connections reaches 94.99% training accuracy, below the single-layer model's 95.25%, indicating training difficulty.A separate ablation replacing hierarchical levels with two word-level BiLSTMs lowers test accuracy to 76.11%, with similar training accuracy and more over-fitting.
- KBQA End-Task Results: The improved relation detector raises the KBQA end task by 2-3% over the baseline relation detector, without joint inference or feature-based re-ranking.Using the top-3 relation detectors yields a new state-of-the-art result on SimpleQuestions and performance comparable to state-of-the-art on WebQSP.
- KBQA End-Task Results: Constraint detection has substantial room to improve when topic-entity and core-chain detection jointly reaches 77.5% top-1 accuracy.The authors contrast this with 58.0% performance for the constraint-detection module.
7 Conclusion
The paper presents HR-BiLSTM for hierarchical matching between questions and KB relations, improving relation detection and supporting state-of-the-art KBQA performance. Future work targets end-to-end integration and more complex QA datasets.
- HR-BiLSTM performs hierarchical matching between questions and KB relations.
- The model outperforms previous methods on KB relation detection tasks.
- Its relation detector enables the KBQA system to achieve state-of-the-art results.
- Future work will integrate HR-BiLSTM into end-to-end systems and evaluate emerging datasets with broader QA characteristics.
Appendix A: Detailed Score Computation for Constraint Detection
The appendix defines character-based lengths and overlap measures for question, entity-name, and entity-mention strings as inputs to score computation.
- Question and entity-name lengths are denoted |q| and |n_e| and measured in characters.
- For an entity mention, the method computes the longest consecutive common subsequence with the entity name.
- The overlap length |m∩e| is measured in characters and used to compute overlap proportions.
- The final question score for a mention linked to entity e is defined from the preceding length and overlap quantities.
Appendix B: Special Rules for Constraint Detection
The appendix describes special constraint-detection rules for dates and answer nodes in WebQSP. These rules adapt matching and filtering to dataset-specific structures.
- Date constraints use threshold θ = 1 because WebQSP questions usually express times as years while KB timestamps use year-month-day formats.
- For answer nodes, constraint detection keeps only constraints based on observed entity types, such as country or city.
Appendix C: Effects of Entity Re-Ranking on SimpleQuestions
The appendix evaluates entity re-ranking on the SimpleQuestions test set and reports that removing this step substantially reduces performance.
- Removing entity re-ranking causes a significant performance drop, while the re-ranker produces especially large gains with beam sizes below 10.
- Table 4 evaluates entity re-ranking on the SimpleQuestions test set.