PaperlayerView exact revision on arXiv

Source-linked AI summary

Interpreting Knowledge Graph Relation Representation from Word Embeddings

Carl Allen, Ivana Balažević, Timothy Hospedales

arXiv:1909.11611v2cs.LGstat.ML

TL;DR

Knowledge graph models achieve strong link prediction with low-rank relation representations, yet the semantic structure they learn is not well explained. The paper derives three relation types and their geometric conditions from PMI-based word-embedding theory, then shows that leading models’ properties and relative performance accord with those conditions.

  • Problem

    The latent low-rank semantic structure learned by knowledge graph representation models remains poorly understood despite improving link prediction performance.

  • Method

    The paper extends theoretical results for PMI-based word embeddings to derive geometric relation conditions that partition knowledge graph relations into three types.

  • Results

    Model properties and relative link prediction performance accord with the derived relation conditions, supporting a common latent structure across word and knowledge graph embeddings.

  • Takeaways & Limitations

    Relation conditions provide a basis for interpreting existing models and selecting models according to relation type.

  • Takeaways & Limitations

    The analysis relies on conditions derived from PMI statistics even though knowledge graph and text embedding data differ significantly.

Abstract

from arXiv · show

Many models learn representations of knowledge graph data by exploiting its low-rank latent structure, encoding known relations between entities and enabling unknown facts to be inferred. To predict whether a relation holds between entities, embeddings are typically compared in the latent space following a relation-specific mapping. Whilst their predictive performance has steadily improved, how such models capture the underlying latent structure of semantic information remains unexplained. Building on recent theoretical understanding of word embeddings, we categorise knowledge graph relations into three types and for each derive explicit requirements of their representations. We show that empirical properties of relation representations and the relative performance of leading knowledge graph representation methods are justified by our analysis.

1 INTRODUCTION

Knowledge graph models learn low-dimensional entity and relation representations for link prediction, but the latent semantic structure underlying their success remains poorly understood. This work transfers theoretical insights from PMI-based word embeddings to derive relation conditions and explain model behavior.

  • 1 INTRODUCTION: Knowledge graph models compress entities into low-dimensional embeddings that support recalling known facts and inferring unknown ones through relation-specific mappings.These models represent relations as transformations from subject embeddings toward related object embeddings.
  • 1 INTRODUCTION: The framework is intended to clarify the latent structure learned by knowledge graph models and guide future model design.The motivation is to explain improved predictive performance rather than merely introduce another scoring function.
  • 1 INTRODUCTION: Recent theory shows that similarity, relatedness, paraphrase, and analogy appear as linear geometric relationships between PMI-vector rows and their low-rank word-embedding projections.The relationships hold subject to known error terms.
  • 1 INTRODUCTION: The paper derives geometric relation conditions from PMI-based word embeddings, partitions knowledge graph relations into three types, and uses them as blueprints for representation models.These conditions specify mappings from subject embeddings to all related object embeddings.
  • 1 INTRODUCTION: The analysis predicts that per-relation ranking and classification performance track how closely models satisfy the appropriate relation conditions.It also links relation properties, such as relatedness strength, to properties of relation matrices.

2 BACKGROUND

Knowledge graph models transform subject embeddings toward objects using additive, multiplicative, or combined score functions, while word embeddings provide the PMI-based geometric foundation for analyzing semantic relations. These representations preserve relational geometry even though their individual dimensions are not directly interpretable.

  • Knowledge graph representation: Knowledge graph models represent entities as low-dimensional vectors and relations as transformations, with score functions differing in transformation form and proximity measure.Examples include matrix multiplication, vector addition, dot products, and Euclidean distance.
  • Knowledge graph representation: Additive models translate subject embeddings and typically use Euclidean distance, with generic score φ(es, r, eo)=−∥es+r−eo∥2.TransE is a simple additive example with zero entity-bias terms.
  • Knowledge graph representation: Bilinear models score subject and object embeddings through a relation-specific matrix, while models such as MuRE combine multiplicative and additive components.DistMult uses a diagonal relation matrix and therefore cannot model asymmetric relations; TuckER constructs relation matrices from prototype matrices.
  • Word embedding: Word2Vec and GloVe produce context-free word embeddings, and Word2Vec can be understood as factorizing a shifted PMI matrix of word co-occurrence statistics.This connects neural word embeddings to earlier count-based representations.
  • Word embedding: PMI-based embeddings encode similarity, relatedness, paraphrase, and analogy as linear relationships inherited from high-dimensional PMI vectors.Low-rank projections preserve relative geometry, although arbitrary scaling or rotation removes direct dimension-level interpretability.

3 FROM ANALOGIES TO KNOWLEDGE GRAPH RELATIONS

The paper extends PMI-based word-embedding insights to knowledge-graph relations, treating relations as geometric mappings from subject to related object embeddings. It distinguishes relation types by shared context and context shifts, derives corresponding model requirements, and predicts how these requirements affect representation properties and link-prediction performance.

  • From analogies to relations: Analogy-style vector offsets arise when word pairs share a common context shift, but not every analogy exhibits such a shift.The car–engine and bus–seats example illustrates that an apparent analogy need not correspond to comparable co-occurrence-distribution differences.
  • Relations as mappings between embeddings: Relation representations must map subject embeddings to all related object embeddings, with relation conditions providing necessary geometric requirements rather than sufficient guarantees.These conditions define mappings evaluated through proximity measures such as dot product or Euclidean distance, but false positives can still arise.
  • Relation types: The framework categorises relations as similarity, relatedness, context-shift, and generalised context-shift according to shared features and relation-specific context differences.Generalised context-shifts cover 1-to-many, many-to-1, and many-to-many relations, while large context sets reduce the fixed structure to a shared subspace component.
  • Categorisation completeness: The three relation types are conjectured to form a complete partition because each feature remains unchanged, changes, or is irrelevant for a relation.They correspond respectively to set equality, subset equality, and relation-specific set difference over PMI word features.
  • Predictions for representation models: Relation complexity is predicted to increase from relatedness to specialisation to context-shift, while model performance depends on satisfying the appropriate additive and multiplicative relation conditions.The analysis contrasts theoretically derived loss functions with existing knowledge-graph models to predict identifiable properties and relative performance.
  • Predicted relation properties: For relatedness relations, relation matrices should be highly symmetric, offset vectors should have low norm, and matrix eigenvalues should reflect relatedness strength.These properties connect the geometry of learned relation representations to the rank of the shared relatedness subspace.

4 EVIDENCE LINKING KNOWLEDGE GRAPH AND WORD EMBEDDINGS

The evaluation tests whether relation-condition predictions explain model performance and learned relation properties across WN18RR and NELL-995. Results broadly support type-specific architectural fit, while hierarchical relations remain difficult.

  • 4.1 P1: JUSTIFYING THE RELATIVE PERFORMANCE OF KNOWLEDGE GRAPH MODELS: MuRE performs best overall, especially on WN18RR and on S/C relations requiring both multiplicative and additive components.TuckER has comparable performance on NELL-995, plausibly because of its multi-task learning ability.
  • 4.1 P1: JUSTIFYING THE RELATIVE PERFORMANCE OF KNOWLEDGE GRAPH MODELS: All models perform best on type R relations, while additive-only models perform worst overall because every relation type involves multiplicative relatedness.DistMult can represent type R relations with no inessential additive parameters.
  • 4.1 P1: JUSTIFYING THE RELATIVE PERFORMANCE OF KNOWLEDGE GRAPH MODELS: Hierarchical relations such as hypernym remain difficult for all models despite abundant training data, consistent with their long paths and near-unit hierarchy scores.Training-instance percentage is not a dominant performance factor when relations are not equally representable.
  • 4.2 P2: PROPERTIES OF RELATION REPRESENTATION: Type R relations show higher symmetry and lower additive-vector norms than S/C relations, while MuRE eigenvalue magnitudes are highest for type R.S/C eigenvalue profiles vary more, reflecting variability in the nature and strength of their relatedness components.

5 CONCLUSION

The paper addresses the limited understanding of latent structure in low-rank knowledge graph models by extending PMI-based word-embedding theory to knowledge graph relations. It derives three relation types and shows that architectural conformity predicts link-prediction performance and relation-representation properties.

  • 5 CONCLUSION: PMI-based word-embedding theory yields geometric relation conditions that map subject embeddings to related object embeddings and partition knowledge graph relations into three types.These conditions provide a basis for analysing existing model loss functions and parameter solutions.
  • 5 CONCLUSION: Models whose architectures better satisfy a relation’s conditions perform better at link prediction under both rank-based metrics and accuracy.The analysis generalises theoretical accounts of similarity and analogy to general knowledge graph relations.

A CATEGORISING WN18RR RELATIONS

WN18RR relations were assigned to categories using the paper’s relation-type scheme. The assignments are documented in a dedicated explanatory table.

  • A CATEGORISING WN18RR RELATIONS: Table 7 explains the category assignment for each WN18RR relation.

B CATEGORISING NELL-995 RELATIONS

NELL-995 relations were categorised using the paper’s relation-type scheme. The category assignments and their explanations are documented in Tables 8 and 9.

  • B CATEGORISING NELL-995 RELATIONS: Tables 8 and 9 provide the NELL-995 relation categorisation and explanations for the assignments.

C SPLITTING THE NELL-995 DATASET

The original NELL-995 test set covered only 10 of 200 training relations, so new validation and test splits were created to represent all training relations fairly.

  • The original test set contained only 10 of the 200 relations present in training.
  • New validation and test sets each sampled 10,000 triples from combined original and training data.

D IMPLEMENTATION DETAILS

All models were reimplemented under a shared PyTorch training setup, with common optimization settings and mostly matched embedding dimensions.

  • All algorithms used PyTorch, Adam, binary cross-entropy, a 0.001 learning rate, batch size 128, and 50 negative samples.
  • Entity and relation dimensions were both 200, except TuckER’s relation dimension, which was 30.

E KRACKHARDT HIERARCHY SCORE

The Krackhardt hierarchy score quantifies directed asymmetry in graph reachability, equaling one for directed acyclic graphs and zero for cycles and cliques.

  • The score measures the proportion of node pairs with a directed path one way but not the reverse.
  • A score of one characterizes directed acyclic graphs, whereas zero characterizes cycles and cliques.
  • Its reachability matrix marks whether a directed path exists between each ordered node pair.

G “OTHER” PREDICTED FACTS

The paper provides sampled unknown triples for three WN18RR relations across probability levels, showing predictions from four models and marking true or self-related facts.

  • The tables sample unknown triples for derivationally_related, instance_hypernym, and synset_domain_topic_of across probabilities near 0.4, 0.6, 0.8, and 1.
  • True triples are bolded, while predictions linking an entity to itself are shown in blue.
  • MuREI, DistMult, TuckER, and MuRE each have a table of predicted “Other” facts.
Loading 1909.11611v2…