PaperlayerView exact revision on arXiv

Source-linked AI summary

Probabilistic Logic Neural Networks for Reasoning

Meng Qu, Jian Tang

arXiv:1906.08495v2cs.LGcs.AIstat.ML

TL;DR

Knowledge graph reasoning must recover missing facts while combining efficient embedding methods with uncertain domain rules. pLogicNet uses a Markov Logic Network optimized by variational EM, alternating embedding-based hidden-triplet inference with rule-weight updates. Experiments on multiple benchmark knowledge graphs report effectiveness over competitive baselines, while embedding methods alone remain unable to leverage logic rules.

  • Problem

    Knowledge graph reasoning seeks missing facts, but MLNs are difficult to infer and embedding methods do not leverage domain knowledge encoded by logic rules.

  • Method

    pLogicNet defines a Markov Logic Network over triplets and uses variational EM, with embeddings inferring hidden triplets in the E-step and rule weights updated in the M-step.

  • Results

    pLogicNet significantly outperforms rule-based methods, most embedding methods, and hybrid baselines under most reported comparisons.

  • Takeaways & Limitations

    Combining complementary information from embedding-based and logic-based predictions yields better performance than pLogicNet using either source alone in most cases.

  • Takeaways & Limitations

    Knowledge graph embedding methods used as a comparison do not leverage logic rules, which encode domain knowledge useful in many applications.

Abstract

from arXiv · show

Knowledge graph reasoning, which aims at predicting the missing facts through reasoning with the observed facts, is critical to many applications. Such a problem has been widely explored by traditional logic rule-based approaches and recent knowledge graph embedding methods. A principled logic rule-based approach is the Markov Logic Network (MLN), which is able to leverage domain knowledge with first-order logic and meanwhile handle their uncertainty. However, the inference of MLNs is usually very difficult due to the complicated graph structures. Different from MLNs, knowledge graph embedding methods (e.g. TransE, DistMult) learn effective entity and relation embeddings for reasoning, which are much more effective and efficient. However, they are unable to leverage domain knowledge. In this paper, we propose the probabilistic Logic Neural Network (pLogicNet), which combines the advantages of both methods. A pLogicNet defines the joint distribution of all possible triplets by using a Markov logic network with first-order logic, which can be efficiently optimized with the variational EM algorithm. In the E-step, a knowledge graph embedding model is used for inferring the missing triplets, while in the M-step, the weights of logic rules are updated based on both the observed and predicted triplets. Experiments on multiple knowledge graphs prove the effectiveness of pLogicNet over many competitive baselines.

1 Introduction

Knowledge graph reasoning seeks to predict missing links from observed triplets, but existing logic-based and embedding methods each have important limitations. pLogicNet combines probabilistic logic rules with embedding-based inference and is reported effective across benchmark graphs.

  • Knowledge graphs support applications including question answering, relation extraction, and recommender systems, but their limited coverage leaves missing-link prediction as a fundamental problem.
  • Markov Logic Networks model uncertain first-order logic rules, but inference is difficult and inefficient because triplets form complicated graph structures.Many missing triplets also cannot be inferred by available rules.
  • Knowledge graph embeddings efficiently predict missing triplets from learned entity and relation representations, but do not leverage logic rules encoding domain knowledge.
  • pLogicNet defines a Markov Logic Network over triplets and trains it with variational EM, using embeddings in the E-step and updating rule weights in the M-step.The framework is trained with stochastic gradient descent and evaluated on four benchmark knowledge graphs.
  • The proposed approach targets simultaneous use of first-order logic, uncertainty handling, effective missing-triplet inference, and efficient training.

2 Related Work

Prior work separates symbolic rules from embedding-based reasoning, while hybrid approaches have not fully addressed both efficiency and rule uncertainty. pLogicNet is positioned as a probabilistic combination of these capabilities.

  • Markov Logic Networks combine first-order logic with probabilistic graphical models, but their inference remains difficult and inefficient for densely connected triplets.Their missing-triplet performance can also be limited when rules do not discover the target facts.
  • Knowledge graph embedding methods learn entity and relation representations with scoring functions that capture patterns such as composition, inverse, symmetric, and asymmetric relations.
  • Embedding methods are effective and efficient but cannot leverage logic rules, while recent hybrid methods do not effectively handle rule uncertainty.
  • Reinforcement-learning approaches train agents to search reasoning paths, but the paper describes their performance as not competitive and pLogicNet as easier to train and more effective.
  • The paper focuses on first-order-logic reasoning over knowledge graphs, distinguishing it from related work on semi-supervised node classification, visual dialog, and graph-neural-network logic reasoning.

3 Preliminary

The preliminary formulation represents knowledge-graph facts as binary triplet variables and reviews MLN and embedding-based distributions. MLNs encode domain rules probabilistically, whereas embeddings score triplets from learned representations.

  • A knowledge graph stores relational facts as (h, r, t) triplets, and limited construction coverage makes predicting missing facts a critical task.
  • The problem assigns each possible triplet a binary indicator, treating observed true facts as known variables and inferring labels for the remaining hidden triplets.
  • MLNs define a joint distribution over observed and hidden triplets using first-order-logic potentials, including composition, inverse, symmetric, and subrelation rules.
  • Each logic rule receives a weight to model uncertainty across its possible groundings, and inference targets the posterior distribution of hidden triplets given observed ones.
  • Embedding methods associate entities and relations with vectors and define triplet probabilities through a scoring function, commonly optimized by stochastic gradient descent.
  • The framework overview alternates logic-based prediction and embedding-based annotation: the E-step supplies extra embedding-training data, while the M-step updates rule weights.

4 Model

pLogicNet combines a Markov logic network with a knowledge graph embedding model, training both through variational EM. Its E-step infers hidden triplets with embeddings, while its M-step updates logic-rule weights using observed and predicted triplets.

  • 4 Model: pLogicNet models the joint distribution of observed and hidden triplets with a Markov logic network and trains it using variational EM.The model alternates inference and learning procedures through a variational E-step and an M-step.
  • 4.1 Variational EM: The variational E-step fixes the logic model and updates the embedding-based distribution to reduce its KL divergence from the true posterior.The embedding model uses a Bernoulli distribution parameterized by a triplet scoring function, with a fixed-point condition involving each triplet’s Markov blanket.
  • 4 Model: Because exact inference over hidden triplets is intractable, pLogicNet approximates their posterior with a mean-field variational distribution.Each hidden-triplet variable is inferred independently, and amortized inference parameterizes the distribution with a knowledge graph embedding model.
  • 4.2 E-step: Inference Procedure: The E-step estimates logic-based conditional probabilities from observed triplets and samples from the current embedding distribution for unobserved connected triplets.These conditional probabilities become targets for updating the knowledge graph embedding model, distilling information from logic rules into embeddings.
  • 4.3 M-step: Learning Procedure: The M-step fixes the embedding distribution and updates logic-rule weights by maximizing expected log-likelihood, using pseudolikelihood instead of the partition-function objective.Observed triplets are treated as true, while embedding probabilities supervise hidden triplets; only a small rule-grounded subset of possible hidden triplets is included.
  • 4.4 Prediction: After iterative E- and M-steps, prediction combines embedding-based and logic-based probabilities for hidden triplets included in the hidden set.For triplets outside that set, the embedding probability is retained while the logic-based probability is replaced with 0.5.

5 Experiment

Experiments evaluate pLogicNet on four benchmark datasets using filtered knowledge graph reasoning, comparing it with embedding, rule-based, and hybrid methods. Results also examine rule patterns, inference embeddings, KGE supervision of rules, and convergence.

  • 5.2.1 Comparing pLogicNet with Other Methods: pLogicNet significantly outperforms rule-based methods and surpasses knowledge graph embedding and hybrid methods under most metrics.The reported gains are attributed to embedding-based inference, logic-rule knowledge, and principled uncertainty handling.
  • 5.2.1 Comparing pLogicNet with Other Methods: pLogicNet∗ outperforms pLogicNet in most cases by combining qθ and p_w to estimate hidden-triplet plausibility.The two components capture different and complementary information.
  • 5.2.2 Analysis of Different Rule Patterns: Most rule patterns significantly improve performance over removing logic rules, but their effectiveness varies across datasets.Inverse and symmetric rules are more important on FB15k, whereas composition and subrelation rules are more effective on FB15k-237.
  • 5.2.3 Inference with Different Knowledge Graph Embedding Methods: Logic rules improve all three tested embedding models, while pLogicNet maintains robust performance with TransE, DistMult, or ComplEx inference.The comparison uses FB15k and WN18RR.
  • 5.2.4 Effect of Knowledge Graph Embedding on Logic Rules: As training proceeds, logic rules discover more positive triplets with stable precision, providing supervision for learning rule weights.This evaluates the interaction between the E-step's rule annotations and the M-step's embedding-based updates.
  • 5.2.5 Convergence Analysis: pLogicNet∗ converges in only 2-3 iterations on both FB15k and WN18.Figure 2 plots iteration on the horizontal axis and Hit@1 (%) on the vertical axis.

6 Conclusion

pLogicNet integrates Markov logic networks with knowledge graph embedding methods for efficient knowledge graph reasoning. It uses variational EM to infer hidden triplets and update rule weights.

  • pLogicNet integrates rule-based methods and knowledge graph embedding methods for knowledge graph reasoning.
  • It models all possible triplets with a Markov logic network and optimizes the model using variational EM.
  • In the E-step, a knowledge graph embedding model infers hidden triplets.
  • In the M-step, rule weights are updated using observed and inferred triplets.
  • Experiments show that pLogicNet is effective across multiple knowledge graphs.

7 Appendix

The appendix reports dataset statistics, implementation settings, hyperparameter thresholds, and comparisons with related reasoning methods. Across matched settings, pLogicNet consistently outperforms RUGE and NNE-AER, while its advantage over RotatE is smaller because RotatE already models important rules implicitly.

  • Dataset statistics and benchmark hyperparameter settings are reported in Tables 6 and 7.
  • Implementation: pLogicNet uses a TransE-based variational distribution with distance-based triplet probabilities, self-adversarial negative sampling, and Adam optimization.
  • Implementation: Rule thresholds are 0.1 for inverse and symmetric rules and 0.6 for composition and subrelation rules.
  • Comparisons with related methods: Compared under RUGE and NNE-AER settings, pLogicNet consistently performs better, attributed to dynamically inferred rule uncertainty through Markov logic networks.The RUGE comparison uses ComplEx with inverse and composition rules, while the NNE-AER comparison uses ComplEx with inverse and subrelation rules.
  • Comparisons with related methods: Against RotatE, pLogicNet shows a less significant improvement because RotatE already implicitly models most important rules on the current datasets.The appendix identifies pLogicNet as a general mechanism applicable beyond knowledge graph reasoning to other statistical relational reasoning tasks.
Loading 1906.08495v2…