PaperlayerView exact revision on arXiv

Source-linked AI summary

Neural Probabilistic Logic Programming in DeepProbLog

Robin Manhaeve, Sebastijan Dumančić, Angelika Kimmig, Thomas Demeester, Luc De Raedt

arXiv:1907.08194v2cs.AI

TL;DR

DeepProbLog addresses the gap between deep learning’s perceptual strengths and probabilistic logic’s reasoning capabilities. It extends ProbLog with neural predicates and adapts inference and gradient-based learning for end-to-end training. The framework demonstrates combined symbolic and subsymbolic reasoning, program induction, and probabilistic logic programming, while exact inference remains a scalability boundary.

  • Problem

    Deep learning handles perception but lacks transparent high-level reasoning, while probabilistic logic approaches lack integrated neural perception.

  • Method

    DeepProbLog extends ProbLog with neural predicates and uses aProbLog-derived gradients to optimize probabilistic-program and neural-network parameters.

  • Results

    DeepProbLog demonstrates combined symbolic and subsymbolic reasoning, program induction, probabilistic logic programming, and learning from examples.

  • Takeaways & Limitations

    The framework integrates general-purpose neural networks with probabilistic-logical modeling and reasoning in an end-to-end trainable language.

  • Takeaways & Limitations

    DeepProbLog currently relies on exact inference, which can become prohibitively expensive for large problems and limits applications such as knowledge-base completion.

Abstract

from arXiv · show

We introduce DeepProbLog, a neural probabilistic logic programming language that incorporates deep learning by means of neural predicates. We show how existing inference and learning techniques of the underlying probabilistic logic programming language ProbLog can be adapted for the new language. We theoretically and experimentally demonstrate that DeepProbLog supports (i) both symbolic and subsymbolic representations and inference, (ii) program induction, (iii) probabilistic (logic) programming, and (iv) (deep) learning from examples. To the best of our knowledge, this work is the first to propose a framework where general-purpose neural networks and expressive probabilistic-logical modeling and reasoning are integrated in a way that exploits the full expressiveness and strengths of both worlds and can be trained end-to-end based on examples.

1. Introduction

DeepProbLog addresses the complementary strengths and limitations of deep learning and probabilistic logic by integrating neural networks with probabilistic logic programming. Its neural predicates support end-to-end learning, symbolic reasoning, and tasks such as digit addition and program induction.

  • Deep learning excels at low-level perception, while probabilistic logic approaches excel at high-level reasoning, motivating their integration.
  • DeepProbLog extends ProbLog with neural predicates rather than embedding reasoning inside a complex neural architecture.
  • Neural predicates provide probabilities from neural networks while probabilistic logic supplies a clear training objective based on example probabilities.
  • In MNIST addition, only the digit predicate is learned, while logical rules encode addition and support reuse for arbitrary digit tasks.
  • DeepProbLog combines neural networks with logical reasoning, probabilistic modeling, programming, and learning from examples, including inductive programming.

2. Background

The background introduces logic-programming semantics and neural-network training as the foundations for DeepProbLog. Logic programs derive consequences from rules, while neural models learn parameterized mappings by minimizing loss with gradient-based optimization.

  • Logic programming concepts: A logic program is a finite set of rules, where a rule derives its head when the conjunction of its body literals holds.
  • Logic programming concepts: Queries ask whether an atom is true in a program’s canonical model, with inference also producing substitutions for non-ground queries.
  • Deep learning: A neural network is a flexible nonlinear function whose parameters are adjusted during training toward desired behavior.
  • Deep learning: End-to-end learning jointly trains model components using only raw inputs and final targets.
  • Deep learning: Softmax outputs represent class probabilities, and cross-entropy measures their deviation from one-hot targets.
  • Deep learning: Gradient-based training repeatedly computes outputs and loss, backpropagates gradients, and updates parameters on minibatches.

3. Introducing DeepProbLog

DeepProbLog builds on ProbLog’s probabilistic facts, rules, worlds, and inference by allowing neural networks to determine probabilities for logical choices. Neural expressions are evaluated on inputs and instantiated into ordinary probabilistic facts or annotated disjunctions.

  • ProbLog: ProbLog programs combine ground probabilistic facts with logic rules, and query probabilities are computed from the possible worlds they hold in.
  • ProbLog: A possible world is formed by selecting probabilistic facts and deriving further ground atoms through the program’s rules.
  • ProbLog: The probability of a possible world is the product of the probabilities assigned to the truth values of its probabilistic facts.
  • ProbLog: Annotated disjunctions represent probabilistic alternatives whose head probabilities sum to at most one.
  • DeepProbLog: DeepProbLog extends ProbLog by obtaining random-choice probabilities from external neural-network functions instead of specifying all of them explicitly.
  • Neural predicates: A neural predicate evaluates a network on input terms and uses its normalized output distribution over classes as probabilities for logical outcomes.
  • Neural predicates: Ground neural expressions are converted into ordinary probabilistic expressions by evaluating the neural network on grounded inputs.

4. Inference

DeepProbLog adapts ProbLog inference by evaluating neural annotated disjunctions and neural facts after grounding, while retaining ProbLog’s logical and arithmetic-circuit pipeline.

  • ProbLog inference: ProbLog inference grounds the program, rewrites it into propositional logic, compiles it into an SDD, and evaluates the resulting arithmetic circuit.Grounding selects query-relevant rules; weighted model counting is performed efficiently after knowledge compilation.
  • ProbLog inference: The earthquake example grounds only facts and rules relevant to calls(mary), omitting hears_alarm(john) and substituting mary for the rule variable.The query probability is represented through weighted model counting and evaluated with an arithmetic circuit rather than enumeration.
  • DeepProbLog inference: DeepProbLog changes inference by instantiating ground neural annotated disjunctions and neural facts using forward passes on relevant neural-network inputs.This occurs after grounding, with neural outputs determining the probabilities of the corresponding ground alternatives or facts.
  • DeepProbLog inference: In the MNIST addition example, grounding retains only digits 0 and 1 because larger digits cannot sum to the query value 1.The grounded neural annotated disjunctions are then converted into regular annotated disjunctions before arithmetic-circuit evaluation.
  • DeepProbLog inference: DeepProbLog programs combine a neural digit predicate with a symbolic addition rule that relates recognized digits to their sum.The example program defines digit recognition through a neural predicate and performs addition symbolically.

5. Learning in DeepProbLog

DeepProbLog learns both probabilistic logic-program parameters and neural parameters by differentiating query probabilities through shared arithmetic circuits and neural backpropagation.

  • Learning objective: DeepProbLog uses learning from entailment, adjusting parameters so predicted query probabilities match desired probabilities under a chosen loss function.Most experiments use positive examples with desired success probability p = 1 and minimize average negative log likelihood.
  • Gradient descent in ProbLog: The gradient semiring augments probability values with derivatives, allowing arithmetic-circuit evaluation to compute query gradients for probabilistic parameters.Its tuple’s first element reproduces probability computation, while the second applies derivative rules.
  • Gradient descent for DeepProbLog: DeepProbLog extends gradient descent by optimizing neural parameters alongside probabilistic parameters while retaining the gradient-semiring computation for the logic program.Neural-network outputs act as differentiable abstract parameters, and the chain rule connects query-probability gradients to internal network parameters.
  • Gradient descent for DeepProbLog: For a ground neural fact, the gradient-semiring label uses the neural network’s output as its probability while preserving semantic equivalence.For neural annotated disjunctions, each domain element receives the corresponding neural-network output component.
  • Learning pipeline: The learning pipeline grounds a query, builds an arithmetic circuit, computes gradients, and updates neural and probabilistic parameters; it can also model label noise explicitly.The noisy MNIST addition example introduces a learnable noise fraction and uniformly distributed corrupted labels.

6. Experimental Evaluation

DeepProbLog is evaluated on logical reasoning with deep learning, program induction, and probabilistic learning tasks. Across these experiments, it improves data efficiency, generalization, robustness, and scalability, while requiring regularization and incurring growing exact-inference costs.

  • Logical reasoning and deep learning: DeepProbLog converges faster and achieves higher accuracy than a CNN baseline on single-digit addition, especially with fewer training examples.The baseline must learn 100 combined sums, whereas DeepProbLog’s neural predicate recognizes only 10 individual digits.
  • Logical reasoning and deep learning: The same trained neural predicate transfers to multi-digit addition without further training, while the baseline fails with few examples and learns slowly with more data.DeepProbLog’s accuracy is lower than on single-digit addition because individual-digit errors compound.
  • Logical reasoning and deep learning: Entropy-based regularization prevents the neural predicate from collapsing to the valid but trivial solution that predicts 0 for every digit.With λ = 8, the neural network consistently converges on the correct solution; lower values are unreliable or trivial.
  • Logical reasoning and deep learning: For noisy-label addition, explicitly modeling noise retains 73.2% accuracy with 80% noisy labels and learns the fraction of noisy examples.The unmodeled DeepProbLog model is more tolerant than the baseline but loses accuracy as noise exceeds 0.5.
  • Program Induction: DeepProbLog reaches up to 96.5% accuracy on program induction, matching prior differentiable-Forth results, and achieves 100% on all reported addition training sizes.For sorting, it maintains 100% accuracy beyond the training lengths where ∂4 fails, while also running faster and scaling better.
  • Probabilistic programming and deep learning: With no additional labels, DeepProbLog converges unreliably on the expected probabilistic solution, but 10 additional labeled examples make convergence reliable.Beyond 20 additional labeled examples, the experiments report no further improvement.
  • Probabilistic programming and deep learning: In the probabilistic card experiment, 6 of 10 runs learned the probabilistic parameters and classified all card types with 100% accuracy.The other four runs failed to converge because classes were permuted or multiple classes mapped to one class.
  • Computation time: Exact inference can make grounding and compilation expensive as problem size grows, reaching an average of 1.3 seconds in the largest experiment.In the smallest experiment, grounding and compilation averages 0.01 seconds, compared with 0.002 seconds for neural-network and arithmetic-circuit evaluation.

7. Related Work

DeepProbLog differs from related neuro-symbolic approaches by embedding neural networks within probabilistic logic programming, preserving logical and probabilistic reasoning while supporting learning and program induction.

  • Many neuro-symbolic methods approximate logic with neural networks, but typically do not support probabilistic reasoning or perception and often restrict programs to non-recursive, acyclic forms.
  • DeepProbLog integrates neural networks into probabilistic logic programming rather than encoding logical terms in Euclidean space.
  • Logic as regularization: Logic-as-regularization methods encode logical background knowledge into neural-network weights so learned models retain logical characteristics after explicit logic is removed.
  • Neural architecture templates: Other approaches use logic as a template for constructing neural architectures, whereas DeepProbLog maintains probabilistic logic-programming semantics.
  • Neural execution: Differentiable theorem proving and differentiable logic execution learn subsymbolic representations or compile restricted logic into differentiable functions, while DeepProbLog emphasizes tight neural-symbolic interaction and probabilistic reasoning.
  • Neural program induction: Neural program-induction work includes neural execution, neurally guided symbolic search, and neural program construction, with DeepProbLog using ProbLog as its host language for logical and probabilistic reasoning.

8. Conclusion

DeepProbLog extends ProbLog with neural predicates to integrate neural networks and probabilistic logic programming and train both components end-to-end. Experiments demonstrate combined symbolic-subsymbolic reasoning, program induction, and probabilistic logic programming, while exact inference limits scalability.

  • DeepProbLog extends ProbLog with neural predicates, using aProbLog gradients and gradient descent to optimize probabilistic-program and neural-network parameters jointly.
  • Experiments demonstrate DeepProbLog’s capabilities in combined symbolic and subsymbolic reasoning, program induction, and probabilistic logic programming.
  • Exact inference can become prohibitively expensive for large problems, so the current implementation cannot yet be applied to tasks such as knowledge-base completion.

Appendix A. DeepProbLog Programs

The appendix presents DeepProbLog programs that combine neural predicates with symbolic rules for arithmetic, sorting, program induction, coin comparison, and poker reasoning.

  • MNIST addition: Single-digit MNIST addition classifies two digit images with a neural predicate and computes their sum symbolically.
  • MNIST addition: Multi-digit addition recursively converts lists of digit images into numbers before adding the resulting integers.
  • MNIST addition: Noisy MNIST addition introduces a probabilistic noise fact and represents noisy outputs with a uniform distribution over 0 to 18.
  • Program induction: Forth addition uses neural predicates for result and carry decisions inside a recursive symbolic addition procedure.
  • Program induction: Forth sorting uses a neural swap predicate within symbolic hole, bubble, and recursive bubblesort predicates.
  • Program induction: The WAP sketch uses neural predicates to select permutations, arithmetic operators, swapping, and a second operator, then executes those choices symbolically.
  • Coins and poker: The coins and poker programs combine neural image classification with symbolic comparison, hand evaluation, recursive number construction, and probabilistic community-card selection.
Loading 1907.08194v2…