Source-linked AI summary
ProjE: Embedding Projection for Knowledge Graph Completion
Baoxu Shi, Tim Weninger
TL;DR
Knowledge graph completion needs effective methods for predicting missing links, but existing embedding models can become parameter-heavy and difficult to scale. ProjE uses embedding projection with shared variables and collective ranking losses, achieving strong performance while using fewer parameters and no prerequisite training. Its broader use of complex path information remains future work because it would add complexity.
Problem
Knowledge graph completion methods based on low-dimensional embeddings face model-size and scalability limitations as architectures become more complex.
Method
ProjE projects candidate entities against a combined input embedding, uses shared entity embeddings with a learnable combination, and trains pointwise or listwise ranking objectives.
Results
ProjE variants outperform existing methods in most entity and relationship prediction cases, with listwise variants generally outperforming the pointwise variant.
Takeaways & Limitations
ProjE shows that a relatively simple two-layer model using length-1 paths can outperform more complex models while avoiding prerequisite embeddings and reducing parameter requirements.
Takeaways & Limitations
Using length-2 and length-3 paths could provide richer information but would necessarily add complexity.
Abstract
from arXiv · showhide
With the large volume of new information created every day, determining the validity of information in a knowledge graph and filling in its missing parts are crucial tasks for many researchers and practitioners. To address this challenge, a number of knowledge graph completion methods have been developed using low-dimensional graph embeddings. Although researchers continue to improve these models using an increasingly complex feature space, we show that simple changes in the architecture of the underlying model can outperform state-of-the-art models without the need for complex feature engineering. In this work, we present a shared variable neural network model called ProjE that fills-in missing information in a knowledge graph by learning joint embeddings of the knowledge graph's entities and edges, and through subtle, but important, changes to the standard loss function. In doing so, ProjE has a parameter size that is smaller than 11 out of 15 existing methods while performing $37\%$ better than the current-best method on standard datasets. We also show, via a new fact checking task, that ProjE is capable of accurately determining the veracity of many declarative statements.
1 Related Work
Existing knowledge graph completion models use low-dimensional embeddings but differ in how they transform and combine them. More complex models can improve accuracy while increasing parameter size, training difficulty, and scalability problems.
- Low-dimensional embedding methods learn vector representations for entities and relationships using margin-based pairwise ranking loss.
- TransE models relationships as translations from a head entity to a tail entity on the same embedding plane.
- Later models modify embedding planes or combination operators, including separate relationship-specific transformations and learned composition functions.
- Neural Tensor Networks use relationship-specific variables, producing substantially more parameters and limiting suitability for networks with moderate relationship counts.
- Path-based models incorporate multi-hop information, but their model size grows exponentially with path length and can worsen scalability issues.
- ProjE has fewer parameters than 11 of 15 methods and requires no prerequisite training.
2 Methodology
ProjE treats knowledge graph completion as ranking candidate entities by projecting them onto a target vector formed from input embeddings. Its shared-variable architecture, alternative ranking losses, and candidate sampling reduce model complexity while supporting pointwise or listwise ranking.
- Model Architecture: ProjE ranks candidate entities by projecting them onto a shared target vector produced from the input embeddings.A learnable combination operator creates the target vector, and candidate projections reveal similarity scores.
- Model Architecture: Diagonal entity and relationship weights provide a sparse combination operator that avoids interactions among feature dimensions at the initial combination stage.The operator uses global diagonal matrices and a combination bias.
- Model Architecture: ProjE uses a combination layer and a projection layer, with candidate rows shared from the entity embedding matrix rather than introducing new variables.The architecture combines entity and relationship embeddings before scoring candidate entities.
- Ranking Method and Loss Function: ProjE supports pointwise and listwise ranking by changing activation functions and corresponding loss formulations.The pointwise variant uses sigmoid and tanh, while the listwise variant uses softmax and tanh to score candidates collectively.
- Candidate Sampling: Candidate sampling reduces projection cost by retaining all positive candidates and sampling only a subset of negatives; performance remains relatively consistent at sampling rates as low as 25%.Sampling creates a smaller working set from the entity embedding matrix.
3 Experiments
The experiments evaluate ProjE on entity prediction, relationship prediction, and fact checking using established datasets and metrics. Across prediction tasks, the ProjE variants generally outperform existing methods, while listwise variants usually outperform the pointwise variant.
- 3.1 Settings: ProjE is evaluated on entity and relationship prediction using FB15K, and on fact checking using DBpedia and SemMedDB.The experiments follow procedures, datasets, and metrics established in related work; the fact-checking task uses real-world questions.
- 3.2 Entity and Relationship Prediction: The three ProjE variants outperform existing methods in most entity-prediction and relationship-prediction cases.Table 3 contains fewer models because many existing methods do not perform relationship prediction.
- 3.2 Entity and Relationship Prediction: The listwise and pointwise ranking adaptations do not significantly improve TransE, indicating that their gains are not merely model-independent tricks.The comparison uses the same hyperparameter settings.
- 3.2 Entity and Relationship Prediction: ProjE_listwise and ProjE_wlistwise perform better than ProjE_pointwise in most cases.The listwise variants use softmax to classify candidate entities collectively, whereas the pointwise variant applies sigmoid independently to each candidate.
- 3.3 Additional Experiments: ProjE performance increases steadily and smoothly during training in supplementary stability experiments.The supplement also examines the model’s scaling potential.
- 3.3 Fact Checking: Fact checking treats declarative statements as link-prediction edges and adapts ProjE_wlistwise to use predicate paths between the two entities.The evaluation removes edges with the queried relationship and tests hundreds of variations across seven question types.
4 Conclusions and Future Work
The paper frames ProjE as a ranking-based embedding model that combines known triple components and projects candidate entities onto the resulting vector. It reports strong performance with a simple architecture and identifies richer path information as a direction for future work.
- 4 Conclusions and Future Work: ProjE views knowledge graph completion as ranking candidate entities by projecting them onto a combined embedding of the known triple components.The model optimizes the ranking score vector collectively in its listwise variation.
- 4 Conclusions and Future Work: ProjE uses directly connected, length-1 paths and a relatively simple 2-layer structure while outperforming models with richer parameter or feature sets.The paper also reports that ProjE requires no pre-trained embeddings and has fewer parameters than related models.
- 4 Conclusions and Future Work: Future work includes incorporating RNNs and CNNs and using length-2 and length-3 paths, although richer path features would add complexity.The authors plan to use complex paths to summarize how entities are connected.
A Appendix
The appendix provides a detailed training algorithm for ProjE_wlistwise and reports additional experiments on training stability and scaling potential.
- A Appendix: The supplement gives a detailed algorithm description for ProjE_wlistwise, with ProjE_listwise as a special case.It also presents two additional experiments.
- A Appendix: The additional experiments examine ProjE’s training stability and scaling potential.These experiments extend the main evaluation beyond prediction-task comparisons.
A.1 Training ProjE
Training ProjE_wlistwise constructs corrupted training instances, optionally samples candidates, and updates model parameters using mini-batch losses.
- A.1 Training ProjE: The training set is created by randomly corrupting either the head or tail entity in each training triple.Positive and negative candidates are then generated, with candidate sampling applied when requested.
- A.1 Training ProjE: For each mini-batch, the algorithm calculates the loss and updates the model parameters.This procedure is applied to the newly generated training data.
A.2 Model Stability
ProjE training is stable across early iterations, with the wlistwise variant performing best among the three variants. The training procedure initializes embeddings and combination operators, constructs positive and sampled negative candidates, and updates parameters with regularization.
- Model Stability: ProjE variants stabilize after the first few iterations, with low score variation indicating stable training progress.The stability is attributed to the Adam optimizer.
- Model Stability: ProjE_wlistwise performs best across all tests, followed by ProjE_listwise and ProjE_pointwise.
- Training Procedure: Training initializes entity and relation embeddings, diagonal combination operators, dropout, candidate sampling, and regularization parameters.
- Training Procedure: Each training iteration constructs data from all triples and samples negative candidates for either missing tails or missing heads.
- Training Procedure: Mini-batches compute the loss, apply regularization, and update all parameters during each training loop.The algorithm includes L1 loss and regularization over entity, relation, and combination parameters.
A.3 Candidate Sampling
ProjE_wlistwise remains effective across candidate sampling rates on FB15K. Sampling can reduce the active working set while preserving robust performance across different positive-to-negative training ratios.
- Candidate Sampling: ProjE_wlistwise is evaluated at five candidate sampling rates ranging from 5% to 95% on FB15K.
- Candidate Sampling: All sampling settings except py = 5% outperform the top-3 existing methods in each metric.The figure captions identify dashed lines as the top-3 existing models for each metric.
- Candidate Sampling: The results support using ProjE with a relatively small sampling rate of 25%.The best results are often achieved under the 25% sampling ratio.
- Candidate Sampling: Performance is robust to different positive-to-negative training data ratios, enabling reduced active working sets for very large datasets.