Source-linked AI summary
Graph Convolution over Pruned Dependency Trees Improves Relation Extraction
Yuhao Zhang, Peng Qi, Christopher D. Manning
TL;DR
Relation extraction models need to capture long-range dependencies without losing crucial off-path information or sacrificing computational efficiency. This paper introduces a graph-convolutional architecture with path-centric pruning, achieving state-of-the-art TACRED performance and further gains when combined with sequence models.
Problem
Existing dependency-based models either lose crucial information through aggressive shortest-path pruning or remain inefficient because tree structures are difficult to parallelize.
Method
The paper extends graph convolutional networks for relation extraction and applies path-centric pruning that removes irrelevant tree content while retaining relevant information.
Results
On TACRED, the C-GCN model outperforms all dependency-based models by at least 1.6 F1 and surpasses the PA-LSTM baseline by 1.3 F1.
Takeaways & Limitations
Dependency-based and sequence models have complementary strengths, and prediction interpolation between them further improves state-of-the-art performance.
Takeaways & Limitations
The model depends on correct dependency parses, while existing parsing algorithms often produce imperfect trees, especially when pruning is used.
Abstract
from arXiv · showhide
Dependency trees help relation extraction models capture long-range relations between words. However, existing dependency-based models either neglect crucial information (e.g., negation) by pruning the dependency trees too aggressively, or are computationally inefficient because it is difficult to parallelize over different tree structures. We propose an extension of graph convolutional networks that is tailored for relation extraction, which pools information over arbitrary dependency structures efficiently in parallel. To incorporate relevant information while maximally removing irrelevant content, we further apply a novel pruning strategy to the input trees by keeping words immediately around the shortest path between the two entities among which a relation might hold. The resulting model achieves state-of-the-art performance on the large-scale TACRED dataset, outperforming existing sequence and dependency-based neural models. We also show through detailed analysis that this model has complementary strengths to sequence models, and combining them further improves the state of the art.
1 Introduction
Dependency-based relation extraction captures long-range syntactic relations but faces efficiency and information-loss problems. The proposed graph-convolutional model and path-centric pruning address these issues while achieving strong benchmark results and complementary gains with sequence models.
- Dependency-based models capture long-range syntactic relations that surface forms may obscure, but traditional approaches suffer from sparse features and lexical brittleness.
- Shortest-path models improve computational efficiency but can discard crucial off-path information such as negation.
- The proposed relation-extraction model extends graph convolutional networks to pool information efficiently over arbitrary dependency structures.
- Path-centric pruning removes irrelevant tree content while retaining relevant information, improving the robustness and performance of dependency-based models.
- On SemEval 2010 Task 8 and TACRED, the model outperforms existing dependency-based neural models and achieves a 10–100x speedup over existing tree-based models.On TACRED, it also surpasses a competitive neural sequence baseline and reaches state-of-the-art performance.
- Detailed analysis finds complementary strengths between dependency-based and sequence models, with their combination further improving state-of-the-art performance.
2 Models
The model applies graph convolutions to dependency structures, pools token representations, and combines sentence- and entity-focused representations for relation classification. Contextualization and careful parsing address remaining limitations, while the architecture retains off-path information beyond shortest-path models.
- Graph Convolutional Networks over Dependency Trees: GCNs encode dependency trees through adjacency matrices, letting each node gather and summarize information from neighboring tokens.The model normalizes activations and adds self-loops to reduce degree-related magnitude differences and preserve each node’s own representation.
- Graph Convolutional Networks over Dependency Trees: Matrix multiplications enable parallel batching and GPU computation, with runtime independent of dependency-tree depth.The model treats dependency graphs as undirected and uses shared parameters across edges.
- Encoding Relations with GCN: After L GCN layers, max pooling produces a sentence representation from token states influenced by neighbors within L dependency-tree edges.The architecture also extracts subject and object representations from the final token representations.
- Encoding Relations with GCN: The classifier concatenates sentence, subject, and object representations, then uses an FFNN, linear layer, and softmax to predict relation probabilities.This combines global sentence information with entity-centric information for classification.
- Contextualized GCN: The architecture remains sensitive to missing word-order context and imperfect parse trees, especially when pruning removes information based on incorrect parses.A jointly trained bidirectional LSTM supplies contextualized inputs and substantially improves performance over the original model.
- Contextualized GCN: Unlike shortest-path approaches, the model incorporates off-path dependency information, while retaining more global information than tree-structured models.This design aims to improve robustness by using local dependency-tree patterns together with crucial off-path content.
3 Incorporating Off-path Information with Path-centric Pruning
The paper introduces path-centric pruning to retain relevant off-path information while removing irrelevant parts of dependency trees. Pruning tokens within distance K of the entity path, especially K = 1, balances these goals.
- Dependency information relevant to a relation usually lies in the lowest-common-ancestor subtree, but aggressive pruning can remove crucial context such as negation.The dependency path alone may also omit conjunction information needed to connect an entity with the event establishing the relation.
- The technique is designed to remove irrelevant information without damaging relevant off-path content.
- Path-centric pruning keeps tokens up to distance K from the shortest entity path within the lowest-common-ancestor subtree.K = 0 keeps only the path, K = 1 keeps directly attached nodes, and K = ∞ retains the entire subtree.
4 Related Work
Prior work shows that syntactic structure helps relation extraction, while neural sequence and dependency-based models address different aspects of the task. This paper studies dependency-based models with architectures intended to improve their efficiency, robustness, and complementarity with sequence models.
- Syntactic features and tree- or path-based kernels have traditionally helped classifiers measure similarity and capture relations in long or structurally complex sentences.
- Neural sequence models use CNNs, RNNs, attention, and recurrent architectures to learn distributed representations for relation extraction.
- Neural dependency models capture long-distance relations through shortest paths, recursive subtrees, or other dependency-tree computations.
- On larger datasets, relatively simple sequence models can match or exceed dependency-based models, motivating a closer study of dependency architectures.
- The paper distinguishes its path-centric pruning from related pruning in semantic role labeling by applying it to entity paths and finding K = 1 effective.
5 Experiments
The experiments compare dependency-based and sequence models on TACRED and SemEval under specified evaluation protocols. Results show strong performance from the proposed models, benefits from pruning and contextualization, and complementary errors that make interpolation effective.
- Experimental Setup: The study compares logistic regression, SDP-LSTM, Tree-LSTM, GCN variants, and PA-LSTM on TACRED and SemEval.
- Experimental Setup: TACRED contains over 106k mention pairs across 41 relation types, while SemEval is smaller and uses macro-averaged F1.
- Experimental Setup: TACRED evaluation selects the median dev-F1 model from five runs and uses entity masking and multichannel word, POS, and NER embeddings.
- TACRED Results: Interpolating GCN and PA-LSTM probabilities reaches an F1 score of 67.1, while C-GCN interpolation reaches 68.2.
- SemEval Results: On SemEval, C-GCN outperforms existing dependency-based neural models under both with-mention and mask-mention evaluation.
- Path-centric Pruning: Across dependency-based models, performance peaks at pruning distance K = 1, while using the entire tree is less effective.
6 Analysis & Discussion
The analysis shows that C-GCN benefits from dependency structure and pruning while complementing sequence models. Its behavior reflects both long-range dependency use and selective incorporation of off-path information.
- Ablation study: Removing dependency structure lowers TACRED dev F1 by 3.2, while removing pruning further hurts performance.The ablation study also finds that entity representations and feedforward layers contribute 1.0 F1.
- Complementary strengths: C-GCN achieves at least 60% higher accuracy than PA-LSTM on 847 dev examples, while PA-LSTM leads on 629 examples.These comparisons aggregate accuracy over five independent runs per example.
- Complementary strengths: Dependency-based models handle sentences with farther-apart entities better, whereas sequence models leverage local word patterns despite parsing errors.The authors identify these as competitive advantages of the two model types.
- Model behavior: C-GCN often focuses on the dependency path but sometimes uses off-path information, while ignoring determiners that rarely affect relation prediction.Figure 5 visualizes token contributions to the pooled sentence representation.
- Model behavior: The highest-scoring dependency edges are usually associated with indicative nouns or verbs for each relation.Edges are scored by summing the contribution dimensions of their connected nodes.
- Qualitative examples: Examples illustrate C-GCN leveraging long-range dependencies with pruning, while PA-LSTM leverages nearby lexical cues and can be less affected by attachment errors.The examples contrast C-GCN errors on a birth-country relation with PA-LSTM errors on no-relation predictions.
7 Conclusion
The paper concludes that graph convolution and path-centric pruning support robust dependency-based relation extraction. It also finds complementary strengths between dependency-based and sequence models, and applies pruning beyond its own model.
- The authors report success for a graph convolutional architecture in relation extraction.
- Path-centric pruning removes irrelevant content without ignoring crucial information, improving the robustness of dependency-based models.
- Dependency-based and sequence models have complementary strengths, and the pruning technique can be applied to other dependency-based models.
A.1 Hyperparameters
The experiments use different hidden sizes and GCN depths for TACRED and SemEval, while sharing Universal Dependencies v1 preprocessing.
- TACRED: TACRED models use hidden size 200, two GCN layers, and two feedforward layers.The GCN uses ReLU nonlinearities and max pooling; Tree-LSTM models use two layers.
- SemEval: SemEval models use LSTM hidden size 100 and one GCN layer.Dependency parses, POS tags, and NER annotations come from Stanford CoreNLP.
- Preprocessing: Both datasets use the Universal Dependencies v1 formalism.
A.2 Training
Training uses stochastic gradient descent with gradient clipping and dataset-specific schedules, plus sentence-representation regularization for GCN models.
- Optimization: Training uses SGD with initial learning rate 1.0 and gradient clipping at 5.
- Optimization: TACRED GCNs train for 100 epochs with learning-rate annealing when dev F1 fails to improve.The learning rate is multiplied by 0.9 after such an epoch, beginning at epoch 5.
- Dataset-specific training: SemEval models train for 150 epochs, while Tree-LSTM models use 30 total epochs on TACRED.
- Regularization: Because sentence representations tend to have large magnitude, the model adds a regularization term to each example’s cross-entropy loss.
- Regularization: The regularizer applies l2 regularization to learned sentence representations with β = 0.003.The authors found it more effective than l2 regularization on convolutional weights.
B Comparing GCN models and PA-LSTM on TACRED
On TACRED, GCN models and PA-LSTM capture substantially different correct examples, and the comparison is unlikely to be explained by random seeds. The analysis therefore supports complementary model behavior.
- Model comparison: 1,450 TACRED dev examples favored at least three GCN models over PA-LSTM, while 1,550 favored PA-LSTM by the same margin.These counts come from 22,631 dev examples accumulated over five independent runs.
- Complementarity: The GCN models capture different examples from PA-LSTM, indicating complementary strengths between dependency-based and sequence models.The paper uses this complementarity to motivate combining the model types.
- Qualitative analysis: Figure 8 visualizes additional TACRED dev examples and their pruned dependency trees that the C-GCN predicted correctly.Word shading represents the number of dimensions contributed to hsent during pooling, with punctuation omitted.
- Qualitative analysis: Table 5 identifies the three dependency edges contributing most to classification across different TACRED relations.The table removes edges involving common punctuation, common prepositions, or tokens within the same entities.
- Model comparison: Only 99 and 121 examples exceeded the three-model margin when comparing two groups of PA-LSTM runs.The much lower counts serve as a random-seed reference for the model-class comparison.