PaperlayerView exact revision on arXiv

Source-linked AI summary

Neural-Symbolic Computing: An Effective Methodology for Principled Integration of Machine Learning and Reasoning

Artur d'Avila Garcez, Marco Gori, Luis C. Lamb, Luciano Serafini, Michael Spranger, Son N. Tran

arXiv:1905.06088v1cs.AI

TL;DR

AI’s growing impact has intensified the need for interpretable and accountable systems with principled knowledge representation and reasoning. This survey examines neural-symbolic integration, combining neural learning with symbolic reasoning and representation, and reports a methodology aimed at explainable AI systems. Its scope includes representation, learning, reasoning, historical systems, applications, trends, and challenges.

  • Problem

    The paper addresses the need for principled knowledge representation and reasoning integrated with deep learning to support sound, explainable, and accountable AI systems.

  • Method

    The paper surveys neural-symbolic computing by examining neural learning, symbolic knowledge representation, reasoning, applications, and integration principles.

  • Results

    The survey identifies neural-symbolic integration as a methodology combining robust neural learning with symbolic reasoning and representation for constructing explainable AI systems.

  • Takeaways & Limitations

    Neural-symbolic systems provide comprehensibility and compositionality through logically structured networks integrating learning and reasoning under different logical systems.

  • Takeaways & Limitations

    Exact inference that maximizes satisfiability is NP-hard, motivating approximate-satisfiability mechanisms in some neural-symbolic systems.

Abstract

from arXiv · show

Current advances in Artificial Intelligence and machine learning in general, and deep learning in particular have reached unprecedented impact not only across research communities, but also over popular media channels. However, concerns about interpretability and accountability of AI have been raised by influential thinkers. In spite of the recent impact of AI, several works have identified the need for principled knowledge representation and reasoning mechanisms integrated with deep learning-based systems to provide sound and explainable models for such systems. Neural-symbolic computing aims at integrating, as foreseen by Valiant, two most fundamental cognitive abilities: the ability to learn from the environment, and the ability to reason from what has been learned. Neural-symbolic computing has been an active topic of research for many years, reconciling the advantages of robust learning in neural networks and reasoning and interpretability of symbolic representation. In this paper, we survey recent accomplishments of neural-symbolic computing as a principled methodology for integrated machine learning and reasoning. We illustrate the effectiveness of the approach by outlining the main characteristics of the methodology: principled integration of neural learning with symbolic knowledge representation and reasoning allowing for the construction of explainable AI systems. The insights provided by neural-symbolic computing shed new light on the increasingly prominent need for interpretable and accountable AI systems.

1 Introduction

Neural-symbolic computing responds to concerns about AI accountability and explainability by integrating neural learning with symbolic knowledge representation and reasoning. The survey presents this integration as a principled route to systems that combine complementary capabilities.

  • Motivation: Deep learning has achieved major results, while its expanding impact has prompted concerns about AI accountability and societal effects.These concerns extend across academic, industrial, and popular contexts.
  • Motivation: Researchers therefore seek better understanding of AI principles and integrated reasoning and learning mechanisms that support reliable, accountable, and explainable systems.Valiant frames learning from experience and reasoning from what has been learned as fundamental cognitive abilities.
  • Neural-Symbolic Computing: Neural-symbolic computing combines learning from experience with reasoning from learned knowledge while reconciling symbolic and connectionist AI paradigms.Knowledge is represented symbolically, while the methodology integrates the two paradigms under a principled foundation.
  • Neural-Symbolic Computing: The survey highlights principled integration as a way to address symbolic brittleness, catastrophic forgetting, extrapolation difficulty, and the gap between perception and abstract reasoning.Neural learning and inference under uncertainty can address brittleness, while symbolic knowledge can support learning and higher-level explanation.
  • Survey Scope: The paper reviews neural-symbolic characteristics, symbolic-knowledge representation categories, applications in learning and reasoning, explainability, recent trends, and research challenges.The representation categories are rule-based, formula-based, and embedding-based.

2 Prolegomenon to Neural-Symbolic Computing

Neural-symbolic systems extend integrated learning and reasoning toward richer cognitive models, including temporal, modal, and multiagent reasoning. The section also presents historical neural implementations and describes how knowledge evolves through time.

  • Neural-Symbolic Systems: Neural-symbolic systems combine learning, reasoning, and knowledge extraction in modular architectures seeking compositionality and improved explainability.They have been applied to data science, ontology learning, simulator training, and cognitive learning and reasoning.
  • Historical Foundations: KBANN enabled learning with background knowledge and knowledge extraction, while CILP integrated learning, reasoning, and knowledge extraction in recurrent networks.CILP also provided a theoretical foundation linking logic programming with neural networks.
  • Historical Foundations: CILP neural networks were shown to approximate the fixed-point operator of propositional logic programs with negation, enabling reasoning and learning with backpropagation.This result supported applications using logic programs as background knowledge.
  • Richer Reasoning: Ensembles of CILP networks were shown to compute modal and temporal fixed-point operators and represent possible-world semantics for several logical fragments.These systems learned solutions to distributed, multiagent learning and reasoning problems.
  • Temporal Evolution: Combining temporal logic with modalities represents how agents’ knowledge and learning evolve over time.CTLK tracks knowledge at successive states, supporting analysis of both the current state and its temporal evolution.
  • Richer Reasoning: Modal and temporal reasoning provide richer knowledge representation languages and support more modular deep networks.The passage connects these capabilities with semantically sound cognitive neural computational models.

3 Knowledge Representation in Neural Networks

Knowledge representation maps symbolic knowledge into neural networks so logical calculus can be performed exactly or approximately, supporting knowledge extraction for explanation and reasoning. The survey groups approaches into rule-based, formula-based, and embedding-based representations, while highlighting trade-offs in expressiveness, inference, and computational tractability.

  • Overview: Knowledge representation maps symbolic knowledge to neural networks, enabling exact or approximate logical calculus and subsequent knowledge extraction.This mapping is the foundation of neural-symbolic systems and supports explanation and reasoning from trained networks.
  • Rule-based Representation: Rule-based methods such as KBANN and CILP encode logical implications with perceptron or recurrent-network structures.CILP generalises KBANN using recurrent networks and bounded continuous units, while CILP++ extends the approach through bottom-clause propositionalisation.
  • Formula-based Representation: Formula-based methods use generative symmetric networks to represent weighted formulas, allowing variables to be inferred beyond the consequent-only limitation of discriminative architectures.Penalty logic uses higher-order Hopfield networks, while confidence rules represent propositional formulas in restricted Boltzmann machines and deep belief networks.
  • Representation Trade-offs: Representation approaches differ in practical trade-offs: formula-based inference and learning remain complex, while symmetric-network partition functions may be analytically intractable.Sum-product methods can ameliorate this intractability, but the survey notes that applying the idea remains unclear in some settings.
  • Propositionalisation: Propositionalisation converts first-order knowledge into propositional clauses that fit neural networks efficiently while preserving entailment.Bottom-clause propositionalisation encodes literals as neural-network features with semantic meaning, but function symbols can produce infinitely many ground terms and irrelevant clauses.
  • Tensorisation: Tensorisation embeds first-order symbols into real-valued tensors, representing constants as one-hot vectors and predicates or functions as matrices or higher-order tensors.Logic Tensor Networks extend relational embedding methods to represent more complex first-order formulas, with reasoning performed through matrix computation.

4 Neural-Symbolic Learning

Neural-symbolic learning combines data-driven neural learning with symbolic knowledge, supporting rule induction, prior-knowledge integration, and knowledge transfer. Approaches include bottom-up and top-down construction of logical rules, with scalability remaining an open issue for some methods.

  • Inductive Logic Programming: Inductive logic programming uses neural-symbolic learning to construct logic programs from examples through bottom-up or top-down rule-learning strategies.Bottom-up methods extract specific clauses and generalise them, whereas top-down methods specialise general clauses.
  • Bottom-up Approaches: Bottom-up methods extract clauses from examples before applying generalisation procedures to search for more general rules.CILP++ and CRILP use bottom-clause propositionalisation, integrating extracted clauses with neural models.
  • Bottom-up Approaches: CILP++ and CRILP learn and fine-tune propositional formulas, but propositionalisation can create many long clauses and very large networks.The passage identifies scalable generalisation within neural networks and extrapolation as open research questions.
  • Top-down Approaches: Top-down methods use neural learning and inference to fine-tune and test rules, often replacing logical operations with differentiable computations.Neural Logic Programming uses TensorLog-style matrix operations, with conjunction and disjunction confidence computed by product and sum.
  • Knowledge-guided Learning: Combining logical rules with data can improve generality, performance, and learning efficiency when datasets are small or prior knowledge is available.Symbolic knowledge can be encoded as controlled training parameters or transferred from a related domain to a student network.
  • Knowledge-guided Learning: Neural-symbolic systems can extract high-level abstractions from complex inputs while integrating logical knowledge for learning and reasoning.The approach connects neural processing of images, audio, and text with logical reasoning capabilities.

5 Neural-symbolic Reasoning

Neural-symbolic reasoning integrates neural learning with model-based inference across logic programs, temporal knowledge, relational embeddings, and approximate satisfiability. Because exact satisfiability inference is NP-hard for arbitrary formulas, some systems use differentiable approximations.

  • Reasoning Approaches: Neural-symbolic reasoning prioritises model-based approaches because theorem-proving systems have generally learned less effectively than end-to-end differentiable systems.Model-based approaches have been implemented for several nonclassical logics and forms of human reasoning.
  • Logic-Program Inference: Forward and backward chaining are commonly implemented through feedforward inference in neural-symbolic systems.Forward chaining derives new facts from rule heads, while backward chaining searches from a query through matching rule bodies.
  • Logic-Program Inference: Forward chaining can support online learning and temporal reasoning, including extracting temporal knowledge for reasoning about driving skills.Differentiable clause functions can perform individual forward-chaining steps.
  • Logic-Program Inference: Backward chaining transforms query search into matrix operations by applying soft logic within neural networks.TensorLog implements this approach using neural representations of symbols.
  • Approximate Satisfiability: Exact inference that maximises satisfiability is NP-hard for logic programs with arbitrary formulas.Some neural-symbolic systems therefore approximate satisfiability rather than solving it exactly.
  • Approximate Satisfiability: Tensor Logic Networks approximate best satisfiability through training and make inference efficient with feedforward propagation.This has enabled applications to the Pascal dataset and image understanding.
  • Relational Reasoning: Relational embedding systems answer queries by searching for the highest grounding score, while visual-reasoning networks infer relationships among image objects.Both approaches use neural representations to reason about entities, relationships, or visual features.

6 Neural-symbolic Explainability

Neural-symbolic explainability represents neural-network behaviour through human-readable rules or other symbolic structures. The surveyed approaches include rule extraction, layer- or module-specific extraction, soft-logic rules, natural-language knowledge, and tree-based program representations.

  • Knowledge Extraction: Neural-symbolic systems can represent connectionist-network behaviour as human-readable expressions to address deep-learning explainability concerns.Rule-extraction methods have been evaluated using accuracy, fidelity, consistency, and comprehensibility.
  • Knowledge Extraction: Combinatorial rule-extraction approaches do not scale well to the dimensionality of current neural networks.Recent work therefore investigates modular networks, specific modules, or selected layers for extraction.
  • Knowledge Extraction: Extracting conjunctive clauses from a deep belief network’s first layer can be fast and effective, whereas higher-layer extraction can reduce accuracy.Soft-logic rule extraction offers a less formal but more flexible alternative.
  • Knowledge Extraction: Deep networks can be coupled with sequence models to extract natural-language knowledge for explainability.Relational extraction can also obtain predicates by inference over text using trained embedding networks.
  • Program Synthesis: Neuro-symbolic program synthesis uses neural networks to construct programs incrementally from input-output samples with tree-based symbolic structures.The tree structure provides an avenue for explainability through explicit nodes, symbols, and rules.

7 Conclusions

The paper concludes that neural-symbolic computing provides a principled integration of neural learning with symbolic representation and reasoning. Sound mappings, logical structure, and compositionality support background knowledge, efficient reasoning, and explainable and accountable AI systems.

  • Conclusions: The survey highlights methodological approaches that integrate effective neural learning with sound symbolic knowledge representation and reasoning.Its central principle is a sound mapping between symbolic rules and neural networks.
  • Conclusions: This mapping allows multiple knowledge-representation formalisms to serve as background knowledge for potentially large-scale learning and efficient reasoning.The conclusion presents this as an interplay between neural learning and symbolic reasoning.
  • Conclusions: Logical structure gives neural-symbolic systems comprehensibility and compositionality while supporting integrated learning and reasoning under different logical systems.The paper identifies learning with the sound semantics of diverse logics as an ongoing research direction.
  • Conclusions: The surveyed methodology contributes toward explainable and accountable AI and machine-learning systems and tools.This conclusion follows from integrating learning, symbolic reasoning, and logical semantics.
  • Conclusions: The paper frames representation, learning, reasoning, and explainability as capabilities distributed across surveyed neural-symbolic systems.The comparison table lists these capabilities for CILP++, Penalty Logic, Confidence Rules, and CRILP.
Loading 1905.06088v1…