Source-linked AI summary
Long-tail Relation Extraction via Knowledge Graph Embeddings and Graph Convolution Networks
Ningyu Zhang, Shumin Deng, Zhanlin Sun, Guanying Wang, Xi Chen, Wei Zhang, Huajun Chen
TL;DR
Distance-supervised relation extraction still struggles with long-tail relations because few training examples are available. The paper transfers relational knowledge from head to tail classes using knowledge graph embeddings, GCNs, and coarse-to-fine knowledge-aware attention, outperforming baselines especially on long-tail relations.
Problem
Distance-supervised relation extraction models perform promisingly on common relations but degrade when relations have few training instances, leaving many long-tail relations data-deficient.
Method
The approach combines implicit relational knowledge from knowledge graph embeddings, explicit relational knowledge from GCNs, and coarse-to-fine knowledge-aware attention for relation extraction.
Results
On the NYT dataset, the model outperforms baselines especially for long-tail relations, and achieves the best results among attention-based models.
Takeaways & Limitations
Relational knowledge from KGs and GCNs provides fine-grained, encoder-agnostic knowledge that improves relation extraction for data-poor tail classes.
Takeaways & Limitations
The authors leave combining the method with denoising, rule mining and reasoning, zero-shot relation extraction, and other NLP scenarios for future work.
Abstract
from arXiv · showhide
We propose a distance supervised relation extraction approach for long-tailed, imbalanced data which is prevalent in real-world settings. Here, the challenge is to learn accurate "few-shot" models for classes existing at the tail of the class distribution, for which little data is available. Inspired by the rich semantic correlations between classes at the long tail and those at the head, we take advantage of the knowledge from data-rich classes at the head of the distribution to boost the performance of the data-poor classes at the tail. First, we propose to leverage implicit relational knowledge among class labels from knowledge graph embeddings and learn explicit relational knowledge using graph convolution networks. Second, we integrate that relational knowledge into relation extraction model by coarse-to-fine knowledge-aware attention mechanism. We demonstrate our results for a large-scale benchmark dataset which show that our approach significantly outperforms other baselines, especially for long-tail relations.
1 Introduction
The paper targets distant-supervision relation extraction when long-tail relations have few examples, using relational knowledge to transfer information from head classes. It combines KG embeddings, GCNs, and coarse-to-fine knowledge-aware attention, and reports stronger performance, especially on long-tail relations.
- Motivation: Distant supervision provides large training corpora but performs poorly when relations have only a few training instances.The resulting data covers relations unevenly, leaving many long-tail relations data-deficient.
- Motivation: Nearly 70% of relations in the NYT dataset are long-tail, making limited-example relation extraction a substantial evaluation concern.The paper identifies long-tail relations as important rather than negligible cases.
- Motivation: Semantically related head relations can provide evidence for predicting data-poor tail relations and reduce uncertainty between candidate relations.The paper illustrates this with burial and death location relations under the same entity-type branch.
- Challenges: The approach must both learn useful relational knowledge while avoiding negative transfer from irrelevant relation classes.It also addresses the challenge of integrating that knowledge into existing relation extraction models.
- Approach: The model uses KG embeddings for implicit class relations, GCNs for explicit relational knowledge, and coarse-to-fine attention to combine that knowledge with sentence representations.Convolutional networks encode sentences before knowledge-aware attention forms bag representation vectors.
- Results: On NYT, the model is effective against baselines, particularly for long-tail relations, while relational knowledge improves relation extraction.The authors also report efficient learning of relational knowledge through GCNs.
2 Related Work
Prior relation extraction work addresses annotation cost, distant-supervision noise, and external knowledge, but generally gives limited attention to long-tail relations and their semantic correlations. The paper positions KG-based relational knowledge and GCNs as a distinct approach to structured relation modeling.
- Relation Extraction: Supervised relation extraction requires substantial annotated data, motivating distant supervision for automatic labeling at scale.Distant supervision introduces wrong-labeling noise, which later work addresses with multi-instance learning and neural architectures.
- Relation Extraction: Many prior distant-supervision studies reduce labeling noise or add external information without addressing the effects of long-tail relations.The related-work discussion distinguishes noise mitigation from long-tail modeling.
- Long-tail Relation Extraction: Existing long-tail relation extraction studies treat relations largely in isolation or use hierarchical attention, rather than modeling their rich semantic correlations.The paper contrasts these approaches with relational knowledge among classes.
- Knowledge Graph Embedding: Knowledge graph embedding methods learn low-dimensional representations of entities and relations, but prior relation extraction extensions neglect correlations between relations.The paper presents its work as the first effort, to its knowledge, to use KG-based relational knowledge of classes for relation extraction.
- Graph Convolutional Networks: GCNs generalize convolution beyond one- and two-dimensional spaces and have been applied to relational data and several NLP tasks.The cited literature includes link prediction, semantic role labeling, dependency parsing, and machine translation.
- Graph Convolutional Networks: Related GCN approaches differ in task scope and graph design: they do not necessarily handle long-tail labels, use attention, or operate on the same hierarchy graph.The paper distinguishes its use of GCNs in a hierarchy graph from label-graph approaches.
3 Methodology
The model combines CNN-based sentence encoding with relational knowledge learned from KG embeddings and GCNs, then uses coarse-to-fine knowledge-aware attention to match textual representations with hierarchical relation classes.
- 3.2 Framework: The framework has three components: instance encoding, relational knowledge learning, and knowledge-aware attention.CNNs encode instance semantics; GCN outputs are combined with pretrained KG embeddings for class representations, which guide attention over instances.
- 3.3 Instance Encoder: CNNs or PCNNs encode each entity-pair instance into a continuous sentence representation.The encoder composes word and position embeddings into instance embeddings, and the model is adaptable to other encoder architectures.
- 3.4 Relational Knowledge Learning: The label hierarchy is initialized with KG embeddings and processed by two-layer GCNs to learn explicit relational knowledge.Base-level relation vectors come from pretrained embeddings, while parent nodes can be initialized by averaging child vectors; parent and child edges use distinct parameters.
- 3.5 Knowledge-aware Attention: The method matches sentence vectors to class embeddings rather than learning separate label-specific attention parameters.This retrieval-style formulation uses relational information among class embeddings to support long-tail relations.
- 3.5 Knowledge-aware Attention: Coarse-to-fine attention compares instance embeddings with relation queries at each hierarchy layer and produces layer-specific textual relation representations.The relation queries follow each target relation’s parent chain, while the attention scores represent instance–query similarity or relatedness.
- 3.5 Knowledge-aware Attention: A second attention mechanism combines hierarchy-layer representations before computing the conditional probability of each relation.Layer weighting reflects that different hierarchy levels can contribute differently to relation prediction.
4 Experiments
Experiments on NYT evaluate the proposed model with held-out precision-recall curves and long-tail Hits@K tests. The model performs best among attention-based systems, improves long-tail relation extraction, and uses relational knowledge from KGs and GCNs, although long-tail results remain unsatisfactory.
- Datasets and Evaluation: The NYT dataset contains 53 relations, 522,611 training sentences, and 172,448 test sentences.
- Datasets and Evaluation: Held-out evaluation compares models using precision-recall curves, while long-tail subsets use macro-average Hits@K for relations with fewer than 100 or 200 training instances.
- Overall Evaluation Results: The proposed KATT approach achieves the best results among attention-based models, outperforming systems with denoising schemes and extra information.
- Long-tail Relations: On long-tail subsets, KATT outperforms plain attention and hierarchical attention for both CNN and PCNN models.However, all evaluated methods remain far from satisfactory on these relations.
- Ablation Study: The ablation study distinguishes coarse-to-fine attention, GCNs, and KG initialization as separate components of KATT.
- Case Study: Case studies show KATT assigning correct sentence attention for long-tail relations, including cases with semantically similar data-rich relations and cases without them.
- Visualizations of Class Embeddings: T-SNE visualizations show that GCNs and pretrained KG embeddings bring semantically similar class embeddings closer, while their contributions differ across relations.Some similar embeddings remain far apart, potentially because of sparse hierarchy structure or equal treatment of sibling nodes.
5 Conclusion and Future Work
The approach uses knowledge from data-rich head classes to improve data-poor tail classes, providing fine-grained relational knowledge through KGs and GCNs. Future work targets denoising, better class embeddings, zero-shot relation extraction, and broader NLP applications.
- The approach transfers knowledge from data-rich head classes to improve data-poor tail classes.
- KGs and GCNs provide fine-grained relational knowledge that is effective and encoder-agnostic.
- Future work includes combining the method with recent denoising methods and using rule mining and reasoning to learn better class embeddings.
- The authors also propose applying the method to zero-shot relation extraction and adapting it to other NLP scenarios.