Source-linked AI summary
Improving Knowledge Graph Embedding Using Simple Constraints
Boyang Ding, Quan Wang, Bin Wang, Li Guo
TL;DR
Knowledge graph embedding has increasingly relied on complex scoring models or extra information beyond triples, motivating simpler structural constraints. The paper adds non-negativity for entities and approximate entailment for relations, and reports consistent improvements over strong baselines on benchmark graphs while preserving efficiency and interpretability. Its evaluation excludes Ensemble DistMult for fair comparison.
Problem
Recent KG embedding work has emphasized complex scoring models or information beyond triples, leaving the potential of very simple constraints insufficiently explored.
Method
The method adds non-negativity constraints to entity representations and approximate entailment constraints to relation representations within a ComplEx-based embedding model.
Results
On WordNet, Freebase, and DBpedia, the constraints yield significant and consistent improvements over strong baselines without negative impacts on efficiency or scalability.
Takeaways & Limitations
The constraints improve model interpretability and substantially increase the structuring of the embedding space.
Takeaways & Limitations
The evaluation excludes Ensemble DistMult to facilitate a fair comparison.
Abstract
from arXiv · showhide
Embedding knowledge graphs (KGs) into continuous vector spaces is a focus of current research. Early works performed this task via simple models developed over KG triples. Recent attempts focused on either designing more complicated triple scoring models, or incorporating extra information beyond triples. This paper, by contrast, investigates the potential of using very simple constraints to improve KG embedding. We examine non-negativity constraints on entity representations and approximate entailment constraints on relation representations. The former help to learn compact and interpretable representations for entities. The latter further encode regularities of logical entailment between relations into their distributed representations. These constraints impose prior beliefs upon the structure of the embedding space, without negative impacts on efficiency or scalability. Evaluation on WordNet, Freebase, and DBpedia shows that our approach is simple yet surprisingly effective, significantly and consistently outperforming competitive baselines. The constraints imposed indeed improve model interpretability, leading to a substantially increased structuring of the embedding space. Code and data are available at https://github.com/iieir-km/ComplEx-NNE_AER.
1 Introduction
Knowledge graph embedding represents entities and relations in continuous vectors to simplify manipulation while preserving graph structure. This paper studies simple non-negativity and approximate-entailment constraints, arguing they improve scalability, interpretability, and predictive performance.
- Knowledge graphs represent entities as nodes and relations as typed edges, with each edge encoded as a head-relation-tail triple.
- KG embedding maps entities and relations into continuous vector spaces, while recent work has pursued more complex scoring models or information beyond triples.
- The approach constrains entity representations to be non-negative and relation representations to encode logical entailment regularities approximately.
- Unlike approaches requiring rule grounding, it applies universal constraints directly to individual entity and relation representations without grounding.
- Evaluations on WordNet, Freebase, and DBpedia report significant, consistent improvements over competitive baselines without negative impacts on efficiency or scalability.
- The constraints improve interpretability and produce a substantially more structured embedding space.
2 Related Work
Related work evolved from simple triple-based embedding models toward complex scoring functions and additional information, while logical-background approaches often require costly rule grounding. This paper instead uses universal, grounding-free constraints and studies non-negativity for interpretable representations.
- Early KG embedding methods learned entity and relation representations from observed triples using simple models such as TransE and RESCAL.
- Later approaches mainly developed more complicated triple scoring models or incorporated information such as entity types, relation paths, and textual descriptions.
- Most logical-background approaches require grounding first-order rules, which can be time- and space-inefficient for complicated rules.
- The proposed constraints apply directly to entity and relation representations, require no manual effort, and are applicable to almost all knowledge graphs.
- Prior work associates non-negativity with sparsity and better interpretability, motivating its investigation for knowledge-graph entity representations.
3 Our Approach
The approach extends ComplEx with non-negativity constraints for entity representations and approximate entailment constraints for relations. These constraints encode interpretability and logical regularities while maintaining comparable complexity.
- Basic embedding model: ComplEx represents entities and relations as complex-valued vectors and scores triples so higher values indicate greater likelihood of truth.
- Non-negativity constraints: Entity representations are constrained to the hypercube [0, 1]^d, storing positive properties and encouraging sparsity and interpretability.
- Approximate entailment constraints: Approximate entailment models ordered relation pairs such as BornInCountry and Nationality with confidence-weighted constraints and slackness.
- Overall model: The model combines ComplEx, bounded non-negative entity representations, and penalized approximate-entailment constraints in one optimization objective.
- Optimization and complexity: The implementation projects entity components into [0, 1]^d after gradient steps, while the approach retains ComplEx-level space and comparable time complexity.
4 Experiments and Results
Experiments evaluate constrained ComplEx on link prediction across WN18, FB15K, and DB100K, alongside analyses of representation structure. The constraints improve predictive performance and make entity and relation embeddings more structured.
- Datasets: The experiments cover WN18, FB15K, and the newly constructed DB100K dataset, with separate training, validation, and test splits.
- Datasets: Approximate entailments are mined from each training set using AMIE+ and PCA confidence above 0.8, yielding 17, 535, and 56 entailments respectively.
- Evaluation: Link prediction ranks candidate entities under the filtered protocol and reports MRR and HITS@N for N = 1, 3, and 10.
- Predictive results: ComplEx-NNE and ComplEx-NNE+AER perform better than or at least as well as ComplEx on almost all metrics across all three datasets, with most improvements statistically significant except on WN18.
- Representation analysis: Non-negativity produces entity representations with higher semantic purity, while same-type entities activate shared dimensions and different-type entities activate distinct dimensions.
- Representation analysis: Relation visualizations organize entailment pairs into equivalence, inversion, and other classes for inspection in the learned embedding space.
5 Conclusion
The paper studies simple constraints that improve knowledge graph embeddings by structuring entity and relation representations. The constraints encode interpretability and logical regularities without substantially increasing space or time complexity.
- Non-negativity constraints target compact, interpretable entity representations, while approximate entailment constraints encode logical regularities in relation representations.
- The constraints impose prior beliefs on embedding-space structure without significantly increasing space or time complexity.
A.1 Sufficient Condition for Eq. (3)
Under non-negative entity representations, strict entailment constraints on relation representations provide a sufficient condition for the model’s entailment inequality.
- Non-negative entity components together with ordered real relation components and equal imaginary components guarantee the entailment inequality.
A.2 Equivalence between Eq. (7) and Eq. (8)
The appendix rewrites the constraints in Eq. (7), simplifies their optimal auxiliary variables, and obtains the optimization formulation in Eq. (8).
- The two constraints in Eq. (7) are first rewritten as a single constraint.
- A second pair of constraints is likewise reduced to one constraint.
- The objective minimizes 1⊤(α+β), determining an optimal value for this term over possible α and β.
- Substituting that optimal value and removing degenerated constraints yields the optimization problem in Eq. (8).
A.3 Analyses on Imaginary Components of Entity Representations
The paper analyzes imaginary entity-representation components using visualizations and entropy measurements across ComplEx-based models.
- Figure 4 visualizes imaginary components learned by ComplEx and ComplEx-NNE+AER using configurations selected by link prediction.
- Figure 5 compares average entropy across imaginary-component dimensions for ComplEx, ComplEx-NNE, and ComplEx-NNE+AER as K varies.
A.4 Properties of Equivalence, Inversion, or Ordinary Entailment
The appendix specifies representation constraints for ordinary entailment, equivalence, and inversion, alongside figures describing entity-component visualizations and entropy comparisons.
- Relation properties: For ordinary entailment rp →rq, the constraints of Eq. (4) directly determine the corresponding relation-representation condition.
- Relation properties: For equivalence rp ↔rq, both directional entailments are represented through a paired constraint condition.
- Entity representations: Figure 4 displays imaginary entity-representation values for four entity groups, ranging from 0 through 0.5 to 1.
- Entity representations: Figure 5 compares average entropy over imaginary-component dimensions for ComplEx, ComplEx-NNE, and ComplEx-NNE+AER as K varies.
- Relation properties: An inverse relation rk^-1 can be represented as the conjugate of relation rk.