Source-linked AI summary
Graph Neural Networks with Generated Parameters for Relation Extraction
Hao Zhu, Yankai Lin, Zhiyuan Liu, Jie Fu, Tat-seng Chua, Maosong Sun
TL;DR
Multi-hop relational reasoning is difficult to apply to natural-language text because existing GNNs operate on predefined graphs. GP-GNNs generates propagation parameters from text, and the paper reports improved relation extraction across three test sets.
Problem
Existing GNNs support multi-hop reasoning on predefined graphs, while enabling multi-hop relational reasoning in natural-language text remains an open problem.
Source: §1 · p. 1Method
Results
GP-GNNs outperforms other models on relation extraction across three test sets and its multi-hop reasoning discovers more relations than baseline models.
Source: §1 · p. 2Takeaways & Limitations
The paper demonstrates multi-hop relational reasoning for relation extraction and presents three datasets for model comparisons across settings.
Source: §1 · p. 2The subject and object embeddings could also carry type information by changing annotators, but this extension is left for future work.
Source: §4.2 · p. 4
Abstract
from arXiv · showhide
Recently, progress has been made towards improving relational reasoning in machine learning field. Among existing models, graph neural networks (GNNs) is one of the most effective approaches for multi-hop relational reasoning. In fact, multi-hop relational reasoning is indispensable in many natural language processing tasks such as relation extraction. In this paper, we propose to generate the parameters of graph neural networks (GP-GNNs) according to natural language sentences, which enables GNNs to process relational reasoning on unstructured text inputs. We verify GP-GNNs in relation extraction from text. Experimental results on a human-annotated dataset and two distantly supervised datasets show that our model achieves significant improvements compared to baselines. We also perform a qualitative analysis to demonstrate that our model could discover more accurate relations by multi-hop relational reasoning.
1 Introduction
GP-GNNs adapts graph neural networks to relational reasoning over text, addressing the gap between graph-based multi-hop reasoning and unstructured language. The paper reports stronger relation-extraction results and introduces datasets for evaluation.
1 Introduction
- Existing GNNs support multi-hop reasoning on predefined graphs but cannot directly perform relational reasoning over natural-language text.The motivating example requires inferring that Luc Besson speaks English from his relation to a film and the film’s language.Source: §1 · p. 1
- GP-GNNs adapts GNNs to text by generating propagation parameters from natural-language information and passing relational messages among entities.Its framework includes encoding, propagation, and classification modules.Source: §1 · pp. 1–2
- GP-GNNs outperforms other models on relation extraction across three test sets and can discover more relations through more robust multi-hop reasoning.The evaluation includes a human-annotated test set and two distantly labeled test sets with different densities.Source: §1 · pp. 1–2
- The paper presents three datasets to help researchers compare relation-extraction models across different settings.The datasets include a human-annotated test set and two distantly labeled test sets.Source: §1 · p. 2
2 Related Work
Prior GNN work spans graph tasks and some natural-language applications, but often relies on predefined graphs. Relation-extraction methods also leave multi-hop patterns among sentence entities underused.
2.1 Graph Neural Networks (GNNs)
- Prior GNN applications to natural-language tasks still perform message passing on predefined graphs, while this paper targets relation extraction from real-world datasets.Earlier GNN work covers tasks including semantic role labeling, relation extraction, and multi-hop question answering.Source: §2.1 · p. 2
2.2 Relational Reasoning
- Relational reasoning has also been studied in visual and physical-object settings, including object interaction and scene-graph construction.Examples include reasoning about picture objects, building scene graphs from images, and modeling physical-object interactions.Source: §2.2 · p. 2
- Existing relation-extraction approaches do not fully use multi-hop inference patterns among multiple entity pairs and their relations within a sentence.The figure illustrates how links involving Léon can support inferring a relation between Luc Besson and English.Source: §2.2 · p. 2, Figure 1 · p. 3
3 Graph Neural Network with Generated Parameters (GP-GNNs)
GP-GNNs builds a fully connected graph over entities and uses text-derived transition matrices to propagate information before making predictions from node representations.
3 Graph Neural Network with Generated Parameters (GP-GNNs)
- GP-GNNs builds a fully connected graph whose entity-to-entity edges correspond to text sequences, then applies encoding, propagation, and classification modules.The task framework reasons over text and entities to predict labels for entities or entity pairs.Source: §3 · p. 3
3.1 Encoding Module
- The encoding module converts the text sequence associated with each edge into a transition matrix that supplies propagation parameters.A sequential encoder such as an LSTM, GRU, or CNN can encode the sequence, with embeddings and layer-specific encoding parameters.Source: §3.1 · p. 3
3.2 Propagation Module
- The propagation module updates node representations layer by layer using the generated transition matrix to pass information from neighboring nodes.Initial node embeddings may encode task-related features or be one-hot representations; the update uses a non-linear activation.Source: §3.2 · pp. 3–4, §3 · p. 3
3.3 Classification Module
- The classification module takes node representations as input, produces predictions, and calculates loss against ground-truth labels.The classification loss uses representations from the propagation layers and classification parameters.Source: §3.3 · p. 4
4 Relation Extraction with GP-GNNs
The section formulates relation extraction as identifying relationships between entity pairs in a sentence, then describes encoding, propagation, and classification modules for that task.
4 Relation Extraction with GP-GNNs
- Relation extraction identifies the pairwise relationship between each entity pair in a sentence.Each entity may consist of one or a sequence of tokens.Source: §4 · p. 4
4.1 Encoding Module
- Entity-pair context is encoded by combining word and position embeddings, then passed through a bidirectional LSTM and an MLP.Position markers distinguish the first entity, the second entity, and other tokens.Source: §4.1 · p. 4
4.2 Propagation Module
- Propagation initializes the target entities with subject/object flag embeddings and other entities with zeros, then uses K layers to propagate information.In the densely connected graph, depth is treated as providing more expressive power.Source: §4.2 · pp. 4–5
4.3 Classification Module
- The classification module uses target-pair embeddings to predict relation labels, with cross-entropy as the classification loss.The pair embeddings are converted for classification by an MLP; the passage also specifies element-wise multiplication in that conversion.Source: §4.3 · p. 5
5 Experiments
The experiments assess relation extraction performance, layer depth, and qualitative differences from baseline models across instance-level and bag-level settings.
5 Experiments
- The experiments aim to test relation extraction performance, the effect of layer count, and qualitative differences between the model and baselines.The planned settings include instance-level extraction on a human-annotated test set and bag-level extraction on a distantly labeled set.Source: §5 · p. 5
- Bag-level relation extraction predicts an entity pair's relation by aggregating a bag of instances.This is described as a widely accepted scheme for relation extraction with distant supervision.Source: §5 · p. 5
- The experiments also include a distantly labeled test subset with large numbers of entities and edges.Source: §5 · p. 5
5.1 Experiment Settings
The experiments use human-annotated and distantly labeled test data, compare GP-GNN variants with other models, and select hyperparameters using validation data.
5.1.1 Datasets
- The distantly labeled data is modified to add missing reversed edges and NA labels for entity pairs without relations.The same training set is used across the experiments.Source: §5.1.1 · p. 5
- The human-annotated test set contains 350 sentences and 1,230 triples accepted by all five annotators.Annotators judged whether distant supervision was correct for each entity pair.Source: §5.1.1 · p. 5
- The dense distantly labeled test set requires more than two entities and a ground-truth circle of at least three entities.It contains 1,350 sentences and more than 17,915 triples and 7,906 relational facts.Source: §5.1.1 · p. 5
5.1.2 Models for Comparison
- Comparisons include Context-Aware RE, CNN and PCNN variants, a one-layer LSTM, and GP-GNNs with two or three layers.The two- and three-layer GP-GNN variants are described as capable of two-hop and three-hop reasoning, respectively.Source: §5.1.2 · pp. 5–6
5.1.3 Hyper-parameters
- Hyperparameters are selected on the validation set, including activation, dn, and tied versus untied adjacency weights.The selected settings are reported in Table 1 and used in all experiments.Source: §5.1.3 · p. 6, Table 1 · p. 6
5.2 Evaluation Details
Bag-level evaluation scores relations for an entity pair from a bag of sentences; the results table identified here covers the human-annotated dataset.
5.2 Evaluation Details
- For bag-level evaluation, a relation score for an entity pair is defined using a max-one setting over its bag of sentences.The passage frames bag-level relation extraction as multi-instance learning.Source: §5.2 · p. 6
- Table 2 reports results on the human-annotated dataset.Source: Table 2 · p. 6
5.3 Effectiveness of Reasoning Mechanism
GP-GNN’s best models significantly outperform all baselines on three test sets, while the authors argue that context-relation co-occurrence can introduce topic-related noise.
5.3 Effectiveness of Reasoning Mechanism
- GP-GNN’s best models significantly outperform all baselines on all three test sets, including sentence-level and bag-level relation extraction.Table 3 reports results on the distantly labeled test set.Source: §5.3 · p. 6, Table 3 · p. 7
- Context-Aware RE may increase topic-similar relation scores through co-occurrence modeling without determining whether context relations participate in reasoning.The authors note this mechanism may introduce noise.Source: §5.3 · p. 6
- Even its one-layer version outperforms CNN and PCNN on the three datasets, which the authors tentatively attribute to complex Wikipedia sentences.They cite a similar conclusion from Zhang and Wang (2015).Source: §5.3 · p. 6
5.4 The Effectiveness of the Number of Layers
Three-layer GP-GNNs perform best across the three datasets, with the added reasoning layer showing a larger improvement on the dense subset than on the overall distantly labeled set.
5.4 The Effectiveness of the Number of Layers
- The three-layer version achieves the best results on all three datasets, and its third-layer improvement is larger on the dense subset than on the overall distant set.The authors say this pattern suggests reasoning helps identify relations especially in sentences with more entities.Source: §5.4 · p. 7
- A K-layer model can infer K-hop relations, and precision curves rise as the number of layers grows.The comparison covers models with different layer counts.Source: §5.4 · p. 7
- On the human-annotated test set, the gain from two to three layers exceeds the gain from one to two; the authors suggest bag-level extraction may be easier.They leave sentence-specific variant selection or ensembling for future work.Source: §5.4 · p. 7
5.5 Qualitative Results: Case Study
The case studies illustrate GP-GNN inferring relations through chains of relations, including a prediction absent from Wikidata, while Context-Aware RE makes a topic-similarity error.
5.5 Qualitative Results: Case Study
- In another case, GP-GNN derives Oozham’s Malayalam language relation by chaining original-language and language-spoken relations.The passage describes the learned logic as original language implying language spoken.Source: §5.5 · p. 7
- Figure 3 compares aggregated precision-recall curves across layer counts on the distant and dense test sets, with Context Aware RE included.Source: Figure 3 · p. 8
- GP-GNN applies the rule owned-by followed by located-in implies located-in to infer that BankUnited Center is located in English, a fact absent from Wikidata.The authors contrast this with Context-Aware RE predicting a topic-similar but incorrect territorial relation between Kentucky and Ohio.Source: §5.5 · p. 8
- Context-Aware RE predicts Kentucky shares a border with Ohio because it models co-occurrence of similarly topical relations; GP-GNN rejects it because Ohio and Johnson County have no relation.Source: §5.5 · p. 8
6 Conclusion and Future Work
The conclusion presents GP-GNNs as a way to use natural-language parameters for relational message passing, demonstrating relation prediction over text and bags while pointing to broader unstructured-input applications.
6 Conclusion and Future Work
- GP-GNNs encode natural language as parameters and propagate messages layer by layer to address relational reasoning over language.Source: §6 · p. 8
- The paper demonstrates GP-GNN effectiveness for predicting relations between entities in natural language and at bag level, with more reasoning hops improving relation extraction.Source: §6 · p. 8
- The authors present graph generation from other unstructured inputs, including images, videos, and audio, as a possible broader framework application.Source: §6 · p. 8