Source-linked AI summary
Informed Machine Learning -- A Taxonomy and Survey of Integrating Knowledge into Learning Systems
Laura von Rueden, Sebastian Mayer, Katharina Beckh, Bogdan Georgiev, Sven Giesselbach, Raoul Heese, Birgit Kirsch, Julius Pfrommer, Annika Pick, Rajkumar Ramamurthy, Michal Walczak, Jochen Garcke, Christian Bauckhage, Jannis Schuecker
TL;DR
Machine learning can be limited by insufficient data and by requirements that data-driven models satisfy scientific, regulatory, or security constraints. This paper defines informed machine learning, proposes a taxonomy of knowledge sources, representations, and integration points, and surveys approaches, uncovering frequent methods and main paths through the field.
Problem
Machine learning may perform unsatisfactorily when insufficient training data are available or when models must satisfy natural-law, regulatory, or security constraints.
Method
The paper defines informed machine learning as using data and independently sourced formal prior knowledge, then classifies approaches by knowledge source, representation, and integration into the machine learning pipeline.
Results
The survey identifies three knowledge-source categories, eight knowledge-representation categories, integration approaches spanning the machine learning pipeline, and recurring taxonomy paths.
Takeaways & Limitations
The taxonomy and its main paths provide guidance for users selecting methods to incorporate prior knowledge and baseline methods for researchers.
Takeaways & Limitations
Comparability of knowledge-graph methods is constrained because studies customize template graphs, whose choice can strongly influence performance and running time.
Abstract
from arXiv · showhide
Despite its great success, machine learning can have its limits when dealing with insufficient training data. A potential solution is the additional integration of prior knowledge into the training process which leads to the notion of informed machine learning. In this paper, we present a structured overview of various approaches in this field. We provide a definition and propose a concept for informed machine learning which illustrates its building blocks and distinguishes it from conventional machine learning. We introduce a taxonomy that serves as a classification framework for informed machine learning approaches. It considers the source of knowledge, its representation, and its integration into the machine learning pipeline. Based on this taxonomy, we survey related research and describe how different knowledge representations such as algebraic equations, logic rules, or simulation results can be used in learning systems. This evaluation of numerous papers on the basis of our taxonomy uncovers key methods in the field of informed machine learning.
1 INTRODUCTION
Machine learning succeeds across many domains but can struggle with limited data, trustworthiness constraints, and interpretability needs. The paper frames informed machine learning as a surveyed family of methods that explicitly integrates formal prior knowledge and organizes them through a taxonomy.
- Machine learning has achieved success in pattern recognition, engineering, and science by learning from many examples.
- Insufficient data can prevent purely data-driven approaches from producing well-performing and sufficiently generalized models.
- Purely data-driven models may fail to satisfy natural-law, regulatory, or security constraints and may need greater interpretability and explainability.
- Researchers integrate prior knowledge through formal representations such as logic rules, algebraic equations, knowledge graphs, and physical simulations.These representations can constrain loss functions, encode relations, or enrich training data.
- The paper surveys informed machine learning and introduces a taxonomy spanning knowledge source, representation, and integration into the learning pipeline.It also proposes an abstract concept and describes how algebraic equations, logic rules, and simulation results are used.
- The survey aims to identify common grounds, diversity, and main research directions without covering every paper or all methodological details.
2 CONCEPT OF INFORMED MACHINE LEARNING
The paper defines informed machine learning around validated, formalized prior knowledge that is independent of the learning task and explicitly integrated alongside data. This extends conventional pipelines by adding a distinct knowledge-information flow.
- 2.1 Knowledge: The paper treats knowledge as validated information about relations between entities in particular contexts.It notes that validation may arise from statistical processing, trusted authorities, empirical studies, or scientific experiments.
- 2.1 Knowledge: More formal knowledge representations are easier to integrate into machine learning.Formalization depends on whether information is written, structured, and expressed in a formal language.
- 2.2 Integrating Prior Knowledge into Machine Learning: Prior knowledge is pre-existent, independent of learning algorithms, and represented externally and separately from the learning problem and training data.
- 2.2 Integrating Prior Knowledge into Machine Learning: Informed machine learning learns from a hybrid information source consisting of data and prior knowledge that is formally represented and explicitly integrated into the pipeline.
- 2.2 Integrating Prior Knowledge into Machine Learning: Unlike conventional machine learning, informed machine learning adds an independent prior-knowledge flow to the usual training-data flow.The prior knowledge can be supplied as logic rules, simulation results, or knowledge graphs and can be integrated through clear interfaces.
3 CLASSIFICATION OF APPROACHES
The paper classifies informed machine learning through a systematic literature survey focused on knowledge sources, representations, and integration points. The resulting taxonomy and Sankey paths expose frequent approaches, their connections, and their main goals.
- Method: The classification methodology uses comparative and iterative literature surveying to identify recurring answers to three analysis questions.The questions concern the source, representation, and pipeline location of integrated prior knowledge.
- Taxonomy: The taxonomy organizes approaches by knowledge source, knowledge representation, and knowledge integration, with each dimension spanning elements found in the literature.The framework is based on the authors’ comparative and iterative survey.
- Taxonomy: The Sankey diagram connects taxonomy elements across dimensions, while element sizes and path widths reflect relative paper counts.Darker paths denote main approaches appearing in at least four papers across all dimensions.
- Key insights: The survey identifies scientific, world, and expert knowledge sources, eight representation categories, and integration across all machine-learning pipeline stages.The representation categories include equations, simulations, invariances, logic rules, knowledge graphs, probabilistic relations, and human feedback.
- Frequent approaches: Frequent paths include scientific knowledge represented by equations and integrated into learning algorithms, and world knowledge represented by logic rules and integrated into hypothesis sets.The examples include algebraic equations in loss functions and logic rules in network architectures.
- Frequent approaches: The survey finds four integration goals—data efficiency, accuracy, interpretability, and knowledge conformity—with better performance as the main goal.The authors note that these goals can be interrelated or partially equivalent under statistical learning theory.
4 TAXONOMY
The taxonomy defines knowledge source, representation, and integration as complementary dimensions for describing informed machine learning. It distinguishes source types, formal representation forms, and the pipeline components where knowledge enters.
- Knowledge Source: Knowledge sources range from formal scientific knowledge to everyday world knowledge and more intuitive expert knowledge.The categories form a spectrum and are neither complete nor mutually exclusive.
- Knowledge Representation: Knowledge representation describes how prior knowledge is formally encoded and provides the interface to the machine-learning pipeline.The taxonomy retains representation forms closest to those used in the surveyed literature, even when transformations are possible.
- Knowledge Representation: The taxonomy covers algebraic and differential equations, simulation results, spatial invariances, logic rules, knowledge graphs, probabilistic relations, and human feedback.These categories capture the varied representation types found in the survey.
- Knowledge Representation: Knowledge graphs encode concepts as vertices and relations as edges, while probabilistic relations encode dependencies or distributions among random variables.Human feedback transfers knowledge through direct user–machine interfaces and modalities such as speech or motion capture.
- Knowledge Integration: Knowledge integration specifies where prior knowledge enters the pipeline: training data, hypothesis set, learning algorithm, or final hypothesis.These four components structure the integration approaches identified in the literature survey.
5 DESCRIPTION OF INTEGRATION APPROACHES
The surveyed approaches integrate different knowledge representations at distinct stages of the learning pipeline. Algebraic equations illustrate how prior knowledge can enter losses, constraints, architectures, data generation, or final-hypothesis checks.
- Overview: The detailed survey is organized by knowledge representation and traces each representation from knowledge source to integration method.The presentation starts with the main path entity found in the largest number of papers.
- Algebraic Equations: The main algebraic-equation path begins with scientific knowledge and enters the learning algorithm, although other integration types are possible.The survey also notes algebraic equations can express more intuitive expert knowledge.
- Algebraic Equations: A knowledge-based loss term adds prior-knowledge violation penalties to the objective while retaining the usual label loss and regularization terms.The weights λ_l, λ_r, and λ_k control the respective terms, and L_k can provide label-free supervision because it depends on inputs and the learned function.
- Learning Algorithm: Algebraic equations are integrated into learning algorithms through additional loss terms or constrained optimization formulations.These terms quantify inconsistencies with physical laws, approximation ranges, or monotonicity constraints.
- Hypothesis Set: Equations can be encoded in neural-network architectures by sequencing predefined operations or imposing linear connections that reflect physical constraints.This represents integration into the hypothesis set rather than only modifying the objective.
- Other Integration Paths: Other paths use equations to generate training data, construct physically informed features, or check final predictions for consistency with domain constraints.Final-hypothesis checks can provide an additional performance metric for model comparison.
5.2 Differential Equations
Differential equations are integrated mainly into neural-network hypothesis sets or learning algorithms, while simulation results commonly augment training data or validate final hypotheses.
- Knowledge representation: Differential equations represent scientific knowledge about dynamical systems and are integrated mainly into neural-network models.The survey focuses particularly on neural-network integration of differential equations.
- Learning Algorithm: Integrating a governing differential equation into the loss function trains networks to approximate its solution while promoting physically plausible outputs.This requires evaluating derivatives of network outputs with respect to inputs, for example through automatic differentiation.
- Hypothesis Set: Neural networks can model unknown parameters in differential equations, producing hybrid architectures that combine learned components with analytically derived functional forms.This approach applies when differential equations contain unknown time- and space-dependent parameters.
- Simulation results: Simulation results mainly extend training data, but they are also integrated into hypothesis sets, learning algorithms, and final hypotheses.Simulations can provide synthetic data, influence learning iterations, or validate trained-model results.
- Training Data: Synthetic data can supply simulated inputs, additional targets, or synthetic labels alongside real-world measurements.Synthetic labels are particularly useful when original experiments are expensive; simulation data can also pre-train or tailor model components.
- Hypothesis Set: Low-fidelity simulations can be embedded in knowledge-based neural networks, enabling models to learn mappings toward real observations or higher-fidelity simulations.Simulation feedback can also improve learning iterations, including reinforcement learning when simulation gradients are unavailable.
5.4 Spatial Invariances
Spatial invariances encode world or scientific knowledge in machine-learning systems, most often through the hypothesis set, while virtual examples and invariant features integrate them through training data.
- Knowledge sources: Spatial invariances commonly represent world knowledge or scientific knowledge and most often inform the hypothesis set.Examples include rotation-invariant object recognition and physical invariants such as angular momentum.
- Hypothesis Set: Physical invariances can be embedded directly into neural-network architectures through invariant tensor bases, Lorentz layers, or group-equivariant convolutions.These designs encode symmetries such as Galilean or mass invariance in the model architecture.
- Hypothesis Set: In support vector machines, invariances and locality can be incorporated through suitable kernels or regularizers penalizing decision-function derivatives.The survey also identifies geometric deep learning and group-theoretic convolutions as approaches for neural networks.
- Training Data: Virtual examples and meta-features integrate invariance knowledge into training data, including features designed to be rotationally, reflectionally, and Galilean invariant.Data augmentation through virtual examples is mathematically equivalent to incorporating prior knowledge through a regularizer.
5.5 Logic Rules
Logic rules formalize world, linguistic, and social knowledge and are integrated mainly into hypothesis sets or learning algorithms through symbolic, probabilistic, and semantic-loss approaches.
- Knowledge sources: Logic rules represent knowledge about real-world objects, object relations, linguistic structure, sentiment, and social-network dependencies.The survey identifies world knowledge as the most frequent source of logic rules.
- Integration paths: Logic rules are integrated mainly into the hypothesis set or, alternatively, into the learning algorithm.The survey distinguishes deterministic and probabilistic hypothesis-set approaches from loss-based algorithmic integration.
- Hypothesis Set: Knowledge-Based Artificial Neural Networks construct network architectures by mapping components of propositional rules to neurons and weights.Extensions can output revised rule sets or support first-order logic.
- Hypothesis Set: Probabilistic rule integration uses logic-templating languages to define distributions over random variables, including Markov logic networks and probabilistic soft logic.These frameworks belong to statistical relational learning and translate first-order rules into probabilistic models.
- Learning Algorithm: Learning algorithms often incorporate logic through additional semantic loss terms that augment the objective function.In student-teacher architectures, rules can first inform a teacher and then guide a student through a semantic imitation loss.
5.6 Knowledge Graphs
Knowledge graphs represent structured relations from world and scientific knowledge and can inform every machine-learning pipeline component, most prominently the hypothesis set.
- Knowledge sources: Knowledge graphs represent structured relations among visual entities, everyday concepts, physical systems, biological pathways, and medical diagnoses.World knowledge is the most frequent source identified in the surveyed papers.
- Integration paths: The survey finds knowledge-graph integration in all four machine-learning pipeline components, with the hypothesis set most prominent.Graph knowledge can enter training data, hypothesis sets, learning algorithms, or final hypotheses.
- Hypothesis Set: Graph neural networks encode object- and relation-centric inductive bias, while graph reasoning layers propagate information through a given knowledge graph.These approaches support applications including physical dynamics, object detection, and image classification.
- Hypothesis Set: Attention mechanisms over knowledge graphs enhance features and support conversational text, broader category use with few data points, and knowledge-informed word embeddings.The survey cites graph attention as a way to use related entities or categories during representation learning.
- Training Data: Distant supervision uses graph information to automatically annotate texts for training, with attention-based networks reducing noisy-sample influence.The original approach treated sentences matching related graph entities as training samples.
- Learning Algorithm: Graph-based regularization can make strongly connected variables behave similarly while leaving unconnected variables freer to differ.The survey describes graph-Laplacian regularization for integrating genetic pathway information.
- Final Hypothesis: Knowledge graphs can post-process embeddings, validate semantic segmentation, or refine object-detection probabilities using semantic consistency.These uses integrate graph knowledge into the final hypothesis or trained-model outputs.
5.8 Human Feedback
Human feedback can enter informed machine learning beyond annotation and feature engineering, including directly through learning algorithms and interactive systems. The surveyed approaches span expert feedback, reinforcement learning, active learning extensions, visual analytics, text analysis, and demonstrations.
- Human feedback is less formalized than other knowledge representations and mainly originates from expert knowledge.
- Human feedback can be integrated directly into learning algorithms, especially in reinforcement learning and interactive learning with visual analytics.
- Learning Algorithm: TAMER replaces predefined rewards with human feedback, allowing agents to learn from environmental observations and human knowledge.
- Standard active learning is excluded when human knowledge is used only to generate labels, while newer approaches integrate additional knowledge into the active-learning process.
- Visual analytics lets users provide corrective feedback, modify instances, and adjust data relationships so machine-learning components incorporate human interpretation.
- Text-analysis tools integrate user-created, merged, or refined topics through updated reference matrices and a regularization term.
- Training Data and Hypothesis Set: Expert demonstrations can pre-train deep Q-networks, accelerating learning while supporting interactive learning through simulations.
6 HISTORICAL BACKGROUND
Knowledge integration in machine learning has developed across symbolic and connectionist traditions. Historical examples include knowledge-based neural networks, knowledge-integrated support vector machines, and statistical-mechanistic combinations in geosciences.
- Symbolic AI emphasized reasoning from symbolic knowledge, whereas connectionist AI later emphasized data-driven decisions using neural networks.
- By the 1990s, knowledge bases were used together with training data to construct knowledge-based artificial neural networks.
- During the 2000s, researchers incorporated knowledge into support vector machines when they were a dominant classification paradigm.
- In geosciences, especially weather forecasting, data assimilation has combined statistical and mechanistic models since the 1950s.
7 DISCUSSION OF CHALLENGES AND DIRECTIONS
The discussion organizes challenges and future directions by knowledge representation, highlighting issues in balancing data with prior knowledge, robustness, comparability, scalability, and human-feedback integration.
- Overview: The survey summarizes its main informed machine learning approaches in Table 4 and discusses their challenges and directions according to integrated knowledge representations.Table 4 records taxonomy paths, motivations, approach ideas, challenges, and current or future directions.
- Algebraic equations: Knowledge-based loss terms integrate algebraic equations, but choosing the relative weights of knowledge supervision and data labels remains challenging.Current approaches set hyperparameters for individual loss terms; the paper suggests self-supervised, few-shot, and theoretical approaches as possible directions.
- Differential equations: Physics-informed neural networks constrain model derivatives with differential equations, but solution robustness and real-time sensor-data integration remain open challenges.The authors also call for uncertainty quantification, comparisons with numerical solvers, online learning, and data-assimilation techniques.
- Simulation results: Simulation-generated or augmented data can mismatch real data, motivating domain adaptation, adversarial training, domain randomization, and more sophisticated hybrid systems.The central boundary is the gap between simulated and real data distributions.
- Spatial invariances: Invariant architectures such as group-equivariant and convolutional networks reduce model search spaces, but correct invariance specification and expensive geometric evaluations remain challenging.The paper identifies efficient adaptation of invariant-based models to new scenarios as a future direction.
- Logic rules: Logic rules can be encoded in KBANN architectures, while future work includes automated integration interfaces, neuro-symbolic systems, and learning rules from data.A key practical question is whether architectures developed for shallow networks remain feasible for deep neural networks.
- Knowledge graphs: Knowledge graphs support explicit propagation and attention or implicit relational inductive bias, but graph choice, entity linking, and continuous fact embedding affect practical comparability.The authors suggest standardized graph pools or benchmarks because graph selection can strongly influence results.
- Human feedback: Human feedback is costly, latency-sensitive, and often intuitive rather than formalized, motivating representation transformations that convert judgments into logical rules.The paper also notes that improved interpretability remains only a minor goal in knowledge integration.
8 CONCLUSION
The paper presents informed machine learning as a unified framework for integrating additional prior knowledge into machine learning. Its taxonomy helps users select methods for insufficient training data and more robust models.
- The taxonomy systematically categorizes approaches for explicitly integrating additional prior knowledge into machine learning.
- The framework helps users identify appropriate methods for applying prior knowledge in machine learning.
- Informed machine learning addresses insufficient training data and supports making models more robust.