Source-linked AI summary
Attention Guided Graph Convolutional Networks for Relation Extraction
Zhijiang Guo, Yan Zhang, Wei Lu
TL;DR
Dependency-tree relation extraction must retain relevant structure without losing useful information through rule-based pruning. AGGCNs address this by learning soft attention over full dependency trees, surpassing prior state-of-the-art models by 8% and 6% accuracy on cross-sentence ternary and binary extraction.
Problem
Rule-based pruning may remove important information from full dependency trees, motivating learned selection of relevant structure for relation extraction.
Method
AGGCNs apply end-to-end soft pruning with attention-guided, fully connected graph representations and dense connections over full dependency trees.
Results
8% and 6% accuracy gains over state-of-the-art models occur on cross-sentence multi-class ternary and binary relation extraction, respectively, while TACRED performance is consistently better.
Takeaways & Limitations
AGGCNs achieve state-of-the-art results across relation extraction tasks while learning to distill useful information directly from full dependency trees.
Takeaways & Limitations
The n-ary relation extraction evaluation assumes a fixed number of entities, with three entities in one dataset and two in another.
Abstract
from arXiv · showhide
Dependency trees convey rich structural information that is proven useful for extracting relations among entities in text. However, how to effectively make use of relevant information while ignoring irrelevant information from the dependency trees remains a challenging research question. Existing approaches employing rule based hard-pruning strategies for selecting relevant partial dependency structures may not always yield optimal results. In this work, we propose Attention Guided Graph Convolutional Networks (AGGCNs), a novel model which directly takes full dependency trees as inputs. Our model can be understood as a soft-pruning approach that automatically learns how to selectively attend to the relevant sub-structures useful for the relation extraction task. Extensive results on various tasks including cross-sentence n-ary relation extraction and large-scale sentence-level relation extraction show that our model is able to better leverage the structural information of the full dependency trees, giving significantly better results than previous approaches.
1 Introduction
The paper introduces AGGCNs, which learn end-to-end soft pruning over full dependency trees while using dense connections to improve graph representations. Across cross-sentence and sentence-level relation extraction, AGGCNs outperform prior models, including current state-of-the-art systems.
- Background: Dependency-based models incorporate dependency trees and capture non-local syntactic relations that sequence-based models may obscure (Peng et al., 2017; Zhang et al., 2018).Existing approaches also use pruning strategies to distill dependency information.
- Motivation: Rule-based pruning can remove important information, motivating a model that learns which parts of the full dependency tree to retain or discard.The introduction illustrates this issue with a cross-sentence n-ary example where the key tokens “partial response” would be excluded by pruning.
- Method: Unlike tree-structured models such as Tree-LSTM (Tai et al., 2015), AGGCNs can be efficiently applied over dependency trees in parallel.Dense connections address the difficulty of capturing distant neighborhood information in shallow graph convolutional networks.
- Results: AGGCNs surpass current state-of-the-art models by 8% on multi-class ternary and 6% on binary cross-sentence relation extraction accuracy, and consistently outperform others on TACRED.These results demonstrate effectiveness across cross-sentence and large-scale sentence-level extraction tasks.
- Method: AGGCNs learn a soft-pruning strategy end to end and combine it with dense connections to obtain better graph representations without additional computational overhead.The model is presented as a novel graph convolutional approach that learns to select and discard information.
2 Attention Guided GCNs
AGGCN transforms full dependency trees into multiple attention-guided fully connected graphs, then integrates their representations through densely connected layers for relation classification. This soft-attention design avoids discarding potentially relevant tree information through rule-based pruning.
- Attention guided layer: AGGCN transforms the original dependency tree into multiple fully connected edge-weighted graphs using attention-guided adjacency matrices.The original adjacency matrix initializes node representations, while multi-head attention constructs up to N attention-guided adjacency matrices; the attention-guided layer begins in the second block.
- Dense and combination layers: Each AGGCN block combines attention-guided, densely connected, and linear combination layers to capture local and non-local structural information.Dense connections link each layer to all preceding layers, enabling deeper models; separate densely connected layers process the N attention-guided graphs before their outputs are integrated.
- Attention guided layer: Unlike hard-pruning strategies that may eliminate relevant information, AGGCN learns soft attention over the full dependency tree.Rule-based pruning assigns zero weights to edges outside a selected subtree, whereas AGGCN retains a fully connected structure and learns edge weights.
- Attention guided layer: Multi-head self-attention jointly attends to different representation subspaces to compute edge weights between arbitrary node pairs.Queries and keys use the collective representation from the preceding AGGCN layer, and each attention head produces one attention-guided adjacency matrix.
- Relation classification: For relation classification, AGGCN concatenates max-pooled non-entity token representations with entity representations and feeds them through an FFNN and logistic regression classifier.The sentence representation masks entity tokens before max pooling, while entity representations are obtained separately from the hidden token representations.
3 Experiments
AGGCN is evaluated on cross-sentence n-ary and sentence-level relation extraction, using standard benchmark settings and metrics. Across these tasks, it improves over dependency-based baselines through attention-guided filtering and dense information propagation over full dependency trees.
- Cross-sentence n-ary relation extraction: AGGCN achieves ternary-relation accuracies of 87.1 (Single) and 87.0 (Cross), surpassing GS GLSTM by 6.8 and 3.8 points, respectively.These results outperform all reported baselines for the binary-class n-ary extraction setting.
- Cross-sentence n-ary relation extraction: On harder multi-class n-ary classification, AGGCN remains 8.0 points higher than GS GLSTM for ternary relations and 5.7 points higher for binary relations.The multi-class task evaluates all instances and causes substantial performance degradation for every model.
- Sentence-level relation extraction: C-AGGCN reaches 85.7 F1 on SemEval, outperforming C-GCN at 84.8 despite the dataset having only one-tenth as many instances as TACRED.On TACRED, AGGCN outperforms GCN by 1.1 F1 points, while adding contextual bidirectional LSTM representations motivates the C-AGGCN variant.
- Ablation and tree-structure analysis: Adding either attention-guided or densely connected layers improves GCN performance, with the attention-guided layer contributing more substantially.Dense connections facilitate propagation across large graphs, while attention distills relevant information and filters noise.
- Ablation and tree-structure analysis: C-AGGCN surpasses C-GCN by 1.5 F1 points with K=1, and its full-tree version outperforms all tested pruned-tree versions.This supports combining densely connected and attention-guided layers while retaining the full dependency tree.
- Training-resource analysis: With 80% of the training data, C-AGGCN reaches 66.5 F1, exceeding C-GCN trained on the full dataset.C-AGGCN consistently outperforms C-GCN at matched training-data sizes, with the gap widening as training data increases.
4 Related Work
AGGCNs extend dependency-based relation extraction and graph convolutional networks by learning edge weights end to end instead of removing edges during preprocessing. Unlike GATs, AGGCNs model relatedness among all nodes and build fully connected graphs to capture long-range semantics.
- Relation Extraction: Relation extraction research spans statistical methods, including tree-based and dependency path-based kernels, and newer sequence-based neural models.Earlier methods also used entity cliques and syntactic features in statistical classifiers.
- Relation Extraction: AGGCNs learn a different weight for each dependency-graph edge end to end, unlike approaches that remove edges during preprocessing.Dependency-based predecessors include splitting the graph into two DAGs for tree LSTMs and encoding the whole graph with graph recurrent networks, while AGGCNs retain the full structure and learn edge relevance.
- Graph Convolutional Networks: AGGCNs are closely related to GCNs, which apply filters over each node’s first-order neighborhood.Graph neural network research progressed from arbitrary structured-graph models to more computationally efficient local spectral convolutions before these GCNs.
- Graph Convolutional Networks: Unlike GATs, which attend only to neighboring nodes while preserving network topology, AGGCNs measure relatedness among all nodes and build fully connected graphs for long-range semantics.GATs use masked self-attentional layers to summarize neighborhood states, but AGGCNs differ in both motivation and network structure.
5 Conclusion
AGGCNs achieve state-of-the-art results across various relation extraction tasks by operating directly on full dependency trees and distilling useful information end-to-end.
- AGGCNs achieve state-of-the-art results on various relation extraction tasks.
- Unlike previous approaches, AGGCNs operate directly on the full tree and learn to distill useful information end-to-end.
Supplemental Material · A Case Study
The case study shows AGGCN correctly predicts a sensitivity relation by using an attention-guided layer over the full dependency tree. Attention visualizations indicate that relevant entity and predictive tokens receive attention from other tokens, unlike path-centric pruning alternatives.
- Supplemental Material: The comparison applies path-centric pruning to the dependency trees, following Zhang et al. (2018), with pruned trees shown for K=0 and K=1.Figure 6 presents the two pruning settings for the cross-sentence n-ary relation-extraction instance.
- A Case Study: AGGCN predicts the correct sensitivity relation for an instance involving tumors, an EGFR L858E mutation, and gefitinib.The instance concerns a partial response to the drug gefitinib.
- A Case Study: The model includes an attention-guided layer that distills relevant information from the full dependency tree end to end.This mechanism enables prediction without relying only on a pruned dependency structure.
- A Case Study: Attention visualizations of two heads show that relevant tokens can be attended by other tokens.The visualized attention comes from the attention-guided layer.
- A Case Study: Entity tokens receive attention in the case-study example, indicating that the model focuses on information tied to the participating entities.The passage identifies entity tokens as relevant tokens attended by other tokens.
- A Case Study: Tokens expressing a partial response are also identified as relevant for predicting the correct relation.The phrase “showed a partial response” is given as an example of predictive information.
B Additional Analysis
Additional analyses compare AGGCN with Graph State LSTM across sentence lengths and maximal neighbor counts in cross-sentence n-ary relation extraction. AGGCN consistently outperforms GS GLSTM, remaining stable on short sentences and performing well with fewer neighbors, potentially because attention creates a fully connected graph that promotes information propagation.
- B Additional Analysis: AGGCN consistently outperforms GS GLSTM across sentence lengths and maximal neighbor counts.The comparison uses test-set performance under both settings in cross-sentence n-ary relation extraction.
- B Additional Analysis: AGGCN remains stable on short sentences, whereas GS GLSTM performance drops, indicating stronger use of context information.
- B Additional Analysis: AGGCN performs well with fewer maximal neighbors, possibly because its attention-guided layer converts the graph into a fully connected graph that encourages information propagation.Graphs with more neighbors can facilitate information exchange because highly connected nodes act as “supernodes” (Song et al., 2018b).