Source-linked AI summary
How Does Knowledge Graph Embedding Extrapolate to Unseen Data: A Semantic Evidence View
Ren Li, Yanan Cao, Qiannan Zhu, Guanqun Bi, Fang Fang, Yi Liu, Qian Li
TL;DR
KGE models can predict unseen triple combinations, but prior work offers limited explanation of why they extrapolate or which factors support that ability. The paper identifies three train-set Semantic Evidences, tests them, and builds SE-GNN to model them explicitly. Experiments report state-of-the-art Knowledge Graph Completion performance on FB15k-237 and WN18RR, alongside improved extrapolation.
Problem
Prior KGE work largely focuses on triple-scoring functions and offers limited explanation of why models extrapolate to unseen triples or which factors support it.
Source: §1 · p. 1Method
The paper defines three train-observable Semantic Evidences and proposes SE-GNN, which models each evidence level with neighbor patterns and combines them through multi-layer aggregation.
Source: §1 · p. 2Results
Takeaways & Limitations
Relation–entity co-occurrence, entity paths, and similarity between observed and predicted entities are reported as important evidence for predicting unseen data.
Source: §6 · p. 7The extrapolation setting concerns unseen triple combinations, with all test entities and relations required to occur in training data.
Source: §3.2 · p. 3
Abstract
from arXiv · showhide
Knowledge Graph Embedding (KGE) aims to learn representations for entities and relations. Most KGE models have gained great success, especially on extrapolation scenarios. Specifically, given an unseen triple (h, r, t), a trained model can still correctly predict t from (h, r, ?), or h from (?, r, t), such extrapolation ability is impressive. However, most existing KGE works focus on the design of delicate triple modeling function, which mainly tells us how to measure the plausibility of observed triples, but offers limited explanation of why the methods can extrapolate to unseen data, and what are the important factors to help KGE extrapolate. Therefore in this work, we attempt to study the KGE extrapolation of two problems: 1. How does KGE extrapolate to unseen data? 2. How to design the KGE model with better extrapolation ability? For the problem 1, we first discuss the impact factors for extrapolation and from relation, entity and triple level respectively, propose three Semantic Evidences (SEs), which can be observed from train set and provide important semantic information for extrapolation. Then we verify the effectiveness of SEs through extensive experiments on several typical KGE methods. For the problem 2, to make better use of the three levels of SE, we propose a novel GNN-based KGE model, called Semantic Evidence aware Graph Neural Network (SE-GNN). In SE-GNN, each level of SE is modeled explicitly by the corresponding neighbor pattern, and merged sufficiently by the multi-layer aggregation, which contributes to obtaining more extrapolative knowledge representation. Finally, through extensive experiments on FB15k-237 and WN18RR datasets, we show that SE-GNN achieves state-of-the-art performance on Knowledge Graph Completion task and performs a better extrapolation ability. Our code is available at https://github.com/renli1024/SE-GNN.
1 Introduction
The paper examines why KGE models can predict unseen triple combinations and how to improve that extrapolation. It identifies three train-set Semantic Evidences and proposes SE-GNN to model them explicitly.
1 Introduction
- Existing KGE work emphasizes triple-scoring functions, leaving limited explanation of why models extrapolate or which factors support it.The paper frames these as two questions: how KGE extrapolates and how to design models with better extrapolation ability.Source: §1 · p. 1
- Semantic Evidence comprises relation–tail co-occurrence, head–tail path connections, and similarity between observed and predicted entities.The paper reports experiments on typical KGE methods to verify the evidences’ relevance to extrapolation.Source: §1 · p. 2
- SE-GNN explicitly models each evidence level through corresponding neighbor patterns and combines them with multi-layer GNN aggregation.The authors propose it to make better use of the three evidence levels and obtain more extrapolative representations.Source: §1 · p. 2
- Experiments on FB15k-237 and WN18RR are used to assess the Semantic Evidence concept and SE-GNN method for Knowledge Graph Completion.The introduction presents these experiments as validating both the evidence concept and the model.Source: §1 · p. 2
2 Related Work
The related-work section groups KGE models by scoring and graph-structure approaches, then distinguishes KGE extrapolation from prior neural-network and GNN studies.
2 Related Work
- KGE models are grouped into translational-distance, semantic-matching, and GNN-based families, with examples including TransE, DistMult, and R-GCN.The section characterizes the first two families by distance- and similarity-based scoring, while GNN-based models capture KG structure.Source: §2 · p. 2
- Prior extrapolation studies focus mainly on classification or regression and node or graph classification, whereas KGE predicts among three mutually influencing targets in a matching task.The authors therefore study KGE from a data-relevant and model-independent view, noting its fact interdependencies and data patterns.Source: §2 · p. 2
3 Knowledge Graph Embedding Extrapolation
The section asks why KGE models can predict unseen triple combinations despite training only on observed triples. It proposes that observable semantic links in the training set provide evidence for such predictions and tests whether stronger evidence aligns with better extrapolation.
3.1 Problem Definition
- The extrapolation problem asks why a model trained to score training triples highly can also perform well on unseen triples.Here, unseen data means new triple combinations, not new entities or relations; all test entities and relations occur in training.Source: §3.1 · p. 3, §3.2 · p. 3
3.2 Extrapolate with Semantic Evidences
- The section proposes three training-set semantic links: relation-to-answer co-occurrence, paths from the head to the answer, and similarity to known answers for the query.These relation-, entity-, and triple-level links are termed Semantic Evidence (SE).Source: §3.2 · p. 3
3.3 Experiment Verification
- The evaluation quantifies relation evidence by training triples sharing r and t, and entity evidence by paths from h to t, limited to length ≤2.The test data is divided into evidence ranges; FB15k-237 ranges are evenly sized except for entity-level evidence, whose proportions are about 6:2:2.Source: §3.3 · pp. 3–4
- Triple-level evidence measures similarity between the unseen answer and known training answers for the same query, using entity-neighbor structure as context.The similarity is based on shared neighbor entity-relation pairs.Source: §3.3 · pp. 3–4
- Figure 2 plots Mean Rank against evidence metric value, with each range’s data share shown on the top x-axis.Lower Mean Rank indicates better prediction; 1 is best.Source: Figure 2 · p. 4, §3.3 · p. 4
- Across models, extrapolation improves consistently as semantic-evidence strength increases, with the same pattern reported on FB15k-237 and WN18RR.Models perform well in high-evidence ranges and are limited in low-evidence ranges.Source: §3.3 · p. 4
4 Semantic Evidence aware GNN
SE-GNN explicitly models the three Semantic Evidence levels through corresponding neighbor patterns and combines them with multi-layer aggregation. Its output entity and relation embeddings are passed to a Knowledge Graph Completion decoder for extrapolation.
4 Semantic Evidence aware GNN
- SE-GNN models each Semantic Evidence level with a corresponding neighbor pattern and combines the evidence through multi-layer GNN aggregation.The paper presents this design as a way to obtain more extrapolative knowledge representations.Source: §4 · p. 4, §4.1 · p. 4
4.1 Modeling SEs with Neighbor Pattern
- Relation-level evidence aggregates neighboring relations, entity-level evidence captures paths through neighboring entities, and triple-level evidence considers both neighbor entities and relations.Iterative aggregation accesses longer paths; the triple-level pattern represents triple similarity from the neighbor-structure view.Source: §4.1 · pp. 4–5
4.2 Model Architecture
- Layer-wise aggregation extends beyond one-hop neighborhoods to model deeper interactions among Semantic Evidence components.The resulting embeddings are used for Knowledge Graph Completion prediction.Source: §4.2 · p. 5, Figure 3 · p. 5
- The output embeddings feed a ConvE decoder for prediction, although the paper notes that other Knowledge Graph Completion decoders could be used.Decoder exploration is outside the paper’s focus.Source: §4.2 · p. 6
5 Experiments
SE-GNN achieves the reported best overall extrapolation performance across the two datasets and performs better than ConvE across Semantic Evidence levels and ranges. Ablations report a performance decrease when Semantic Evidence modeling parts are removed.
5.1 Experiment Setup
- Experiments use FB15k-237 and WN18RR, with filtered evaluation and five ranking metrics; baselines span translational, semantic-matching, and GNN-based models.For tied prediction scores, the evaluation uses the average of the upper- and lower-bound ranks.Source: §5.1 · p. 6, §5.2 · p. 6
5.3 Effective Modeling of Semantic Evidences
- SE-GNN performs better than ConvE at every Semantic Evidence level across all evidence ranges.The comparison controls the decoder by using ConvE in both models.Source: §5.3 · p. 6, Figure 4 · p. 7
5.2 Results of Knowledge Graph Completion task
- Across five metrics on two datasets, SE-GNN achieves 9 of 10 SOTAs and is overall best against baselines, with a competitive WN18RR MR result.The exception to the 9-of-10 SOTA results is MR on WN18RR.Source: §5.2 · p. 6, Table 1 · p. 7
5.4 Ablation Study of each Semantic Evidence
- Removing one or two Semantic Evidence modeling parts degrades performance across all six SE-GNN variants.The authors interpret this as evidence for the effectiveness of each modeling part.Source: §5.4 · pp. 6–7, Table 2 · p. 7
6 Conclusion
The paper identifies three train-set Semantic Evidence signals associated with predicting unseen data and reports consistent improvement from SE-GNN across different datasets. It points to richer evidence and more elaborate SE modeling as future directions.
6 Conclusion
- The study identifies relation–entity co-occurrence, entity paths, and similarity between observed and predicted entities as three levels of Semantic Evidence.It reports that quantitative experiments and a qualitative case study verify the effectiveness of these signals.Source: §6 · p. 7
- SE-GNN explicitly models the three evidence levels and achieves consistent improvement on different datasets.Source: §6 · p. 7
- Future work includes exploiting more extrapolative evidence and designing more elaborate SE modeling methods.Source: §6 · p. 7
A Case study of Semantic Evidence
The case study illustrates three forms of Semantic Evidence using unseen predictions on FB15k-237. Across the displayed cases with abundant evidence, the examined KGE models make good extrapolative predictions.
A Case study of Semantic Evidence
- Relation evidence connects a relation with a frequently co-occurring tail: travel-month examples associate December with travel-month queries.The paper presents this co-occurrence as one contributor among three evidence levels.Source: A Case study of Semantic Evidence · p. 8
- Entity evidence uses paths between entities; several training-set connections link Robert Downey Jr. to New York City, supporting a live-in prediction.Source: A Case study of Semantic Evidence · p. 8
- Triple evidence transfers a prediction to a similar entity, illustrated by inferring that a freshman program contains Computer Science from similar observed majors.Source: A Case study of Semantic Evidence · p. 8
- For the displayed unseen cases with abundant Semantic Evidence, all examined KGE models make good extrapolative predictions.The authors describe these cases as verifying the Semantic Evidence concept.Source: A Case study of Semantic Evidence · p. 8
B Correlation between SE and Extrapolation Performance on WN18RR Dataset
On WN18RR, the reported association between stronger Semantic Evidence and extrapolation performance holds across models and evidence types. The paper notes that this dataset’s limited relation set corresponds to a weaker relation-level trend than entity- and triple-level trends.
B Correlation between SE and Extrapolation Performance on WN18RR Dataset
- Because WN18RR has much low-evidence data, evenly dividing its test data into three ranges as in Figure 2 is difficult, so the data are divided into [0, 1) and [1, Max].The ranges respectively represent data with evidence and without evidence; Max denotes the maximum value of the corresponding evidence metric.Source: B Correlation between SE and… · pp. 8–9
- Figure 6 reports that stronger evidence is associated with better extrapolation across models and evidence types on WN18RR.The figure plots Mean Rank across evidence ranges; lower values indicate better performance.Source: Figure 6 · p. 9, B Correlation between SE and… · p. 9
- WN18RR’s 11 relation types correspond to a weaker downward trend for relation-level evidence, while entity- and triple-level evidence play a more important role.Source: B Correlation between SE and… · p. 9
C Dataset Statistics
The experiments use FB15k-237, derived from Freebase, and WN18RR, derived from WordNet. Both remove duplicate and inverse relations, but only FB15k-237 imposes the stated restriction on direct train-set connections for validation and test triples.
C Dataset Statistics
- FB15k-237 is a pruned Freebase dataset that removes duplicate and inverse relations and excludes direct train connections for validation and test triples.Source: C Dataset Statistics · p. 9
- WN18RR is derived from WordNet and pruned to remove duplicate and inverse relations, without FB15k-237’s direct-connection restriction.Source: C Dataset Statistics · p. 9
D Experimental Details
SE-GNN uses inverse edges and relations to support bidirectional information flow and unify prediction directions, while randomly removing training-batch edges during aggregation to reduce leakage and approximate extrapolation.
D Experimental Details
- Adding an inverse edge for each knowledge-graph edge makes the graph undirected to pass information bidirectionally and enhance connectivity.This follows CompGCN (Vashishth et al. 2020b).Source: D Experimental Details · p. 9
- Inverse relations transform tail- and head-prediction into a unified prediction format, improving computational efficiency.This follows ConvE.Source: D Experimental Details · p. 9
- Training-batch edge removal prevents prediction-edge leakage during aggregation and guides learning interactions between observed and missing edges.The authors describe this as a closer scenario of extrapolation.Source: D Experimental Details · p. 9