Source-linked AI summary
Incorporating Domain Knowledge into Deep Neural Networks
Tirtharaj Dash, Sharad Chitlangia, Aditya Ahuja, Ashwin Srinivasan
TL;DR
The paper addresses how human domain knowledge can be incorporated into neural networks for scientific assistants and other human–machine data-understanding settings. It surveys logical and numerical constraint-based approaches and their subcategories. The survey concludes that these approaches provide varied ways to encode domain knowledge, while integration still faces challenges involving differentiability, network structure, explainability, and broader forms of knowledge.
Problem
Neural-network model construction needs a precise way to incorporate human domain knowledge, while existing practice may treat datasets similarly and ignore knowledge beyond raw data.
Method
The paper surveys incorporation of domain knowledge into deep neural networks as logical constraints or numerical constraints, including propositional and predicate logic, loss functions, and weight constraints.
Results
The survey describes techniques for encoding domain knowledge through rule-based structures, relational representations, knowledge graphs, and neural-network constraints across several subcategories.
Takeaways & Limitations
Domain knowledge can be integrated into neural networks through multiple logical and numerical representations, supporting human–machine collaboration in AI for Science.
Takeaways & Limitations
Logical integration is constrained because logic is not differentiable for standard end-to-end gradient training, while propositionalisation can lack variable sharing and involve extremely large feature spaces.
Abstract
from arXiv · showhide
We present a survey of ways in which domain-knowledge has been included when constructing models with neural networks. The inclusion of domain-knowledge is of special interest not just to constructing scientific assistants, but also, many other areas that involve understanding data using human-machine collaboration. In many such instances, machine-based model construction may benefit significantly from being provided with human-knowledge of the domain encoded in a sufficiently precise form. This paper examines two broad approaches to encode such knowledge--as logical and numerical constraints--and describes techniques and results obtained in several sub-categories under each of these approaches.
1 Introduction
The survey addresses how to provide domain knowledge to deep neural networks for human–machine collaboration in AI for Science. It organizes this incorporation into logical and numerical constraints, with the broader goal of making machine-based model construction use precise human knowledge.
- AI for Science uses AI methods to accelerate understanding of the natural world and support engineering, medicine, healthcare, agriculture, and environmental applications.
- Effective scientist-in-the-loop systems require machines to receive human knowledge precisely and communicate their findings understandably.
- The introduction connects domain-knowledge incorporation to a broader AI challenge because standard machine-learning practice may treat datasets alike while ignoring knowledge beyond raw data.
- The survey focuses on incorporating domain knowledge into neural networks with one or more hidden layers, including deep neural networks.
- The paper studies domain knowledge represented as logical constraints and numerical constraints, with subcategories including propositional logic, predicate logic, loss functions, and weight constraints.
2 Domain-Knowledge as Logical Constraints
Logical constraints incorporate domain knowledge by shaping neural-network structure, inputs, or learning through propositional and first-order representations. The surveyed approaches range from rule-derived architectures and propositionalized features to knowledge graphs, relational constraints, and enriched graph inputs.
- Propositional Logic: Propositional rules can constrain neural-network structure, after which weights are learned normally within that structure.This produces weighted forms of rule antecedents.
- Propositional Logic: KBANN converts hierarchically structured propositional rules into a fixed neural-network topology, but its rule-refinement capacity is bounded by the pre-existing rules.
- Propositional Logic: CIL2P translates propositional logic programs into neural networks and supports a correctness result showing a structure that computes the logical consequences of the domain knowledge.
- Binary Relations: Knowledge-infused learning incorporates binary relations from knowledge graphs at shallow, semi-deep, or deep layers using embeddings, attention, constraints, or stratified representations.
- n-ary Relations: Propositionalisation converts relational knowledge into Boolean-valued features for deep networks, including Deep Relational Machines and bottom-clause-based feature vectors.
- n-ary Relations: Propositionalisation lacks variable sharing between first-order features and can require separate feature selection because its possible feature space is extremely large.
- n-ary Relations: Vertex enrichment adds first-order background relations as node features, providing a simplified way to incorporate symbolic knowledge into standard graph neural networks.
3 Domain-Knowledge as Numerical Constraints
The survey describes numerical constraints as a major route for incorporating domain knowledge into neural networks, chiefly through loss functions, priors, transfer learning, and regularisation. These methods add domain-informed structure to training, parameters, or representations.
- Loss functions: Additional loss terms combine task-specific objectives with constraints on inputs, outputs, or the learned function.The weighting parameters determine the relative contribution of the total-loss components.
- Loss functions: Semantic loss measures how well neural-network outputs satisfy propositional rules by penalising low probability of constraint-satisfying states.The loss is proportional to the negative logarithm of the probability of generating a satisfying state.
- Loss functions: Iterative distillation transfers first-order logic rules into neural-network weights while preserving flexibility in the choice of deep-network architecture.The approach modifies a knowledge-distillation loss to integrate structured rule information.
- Priors and transfer learning: Transfer learning constructs a target-domain weight prior from a related source-domain posterior, particularly when target-domain data are scarce.The method can replace and separately fine-tune the final layer for a target task.
- Numerical constraints: Numerical constraints are incorporated through loss functions, constraints on weights, priors, and transfer learning.The survey treats these as sub-categories of numerical constraints, alongside loss functions and regularisation terms.
- Regularisation: Prior knowledge induces regularisation, including L1 and L2 penalties that reduce model complexity and can support deep-network construction with less data.The optimiser minimises the task loss together with the regularisation term.
4 Challenges and Concluding Remarks
The survey identifies practical and conceptual challenges in translating logical or numerical constraints into deep networks. It also highlights unresolved issues around trust, explanations, fairness, and representing domain knowledge beyond formal constraints.
- Practical challenges: There is no standard framework for translating logical constraints into neural networks.Simplification methods can produce representations consumable by standard deep networks, but the translation process has limitations.
- Logical constraints: Logic is not differentiable, preventing standard end-to-end gradient-based training; existing gradient propagation can be intractable for routine use.Logical formulas can also introduce cyclic dependencies, whereas neural networks are directed acyclic graphs.
- Numerical constraints: Constructing numerical loss terms is not straightforward and can create difficult, sometimes constrained, optimisation problems requiring additional mathematical tools.
- Explainability: Deep-network models in human-in-the-loop workflows need to be human-understandable, but explanatory models identifying true causal connections remain elusive.Existing work cited in the survey generates post hoc explanations that are locally consistent.
- Fairness and ethics: Domain knowledge may help address bias declaratively through constraints or through ethical penalty terms, but transfer of these approaches to deep-network construction remains under investigation.
- Beyond formal constraints: Language models raise the possibility of representing domain knowledge in forms beyond logical or numerical constraints, although accessible informal-knowledge assistants still require further development.Formal precision may remain necessary for scientific assistants, while its role in commonsense representation is less evident.