PaperlayerView exact revision on arXiv

Source-linked AI summary

Logic Explained Networks

Gabriele Ciravegna, Pietro Barbiero, Francesco Giannini, Marco Gori, Pietro Lió, Marco Maggini, Stefano Melacci

arXiv:2108.05149v1cs.LGcs.AIcs.LOcs.NE

TL;DR

Neural networks often make decisions without human-understandable explanations, a concern in safety-critical settings. This paper introduces LENs, neural models that use human-understandable concepts to produce FOL explanations. Across considered tasks, µ and ReLU LENs lie on the accuracy–interpretability Pareto frontier, while LEN explanations are less complex than BRL rules in the reported comparisons.

  • Problem

    Neural networks’ decision processes can be opaque, while human-understandable explanations are important in safety-critical settings.

    Source: §1 · pp. 1–2

  • Method

    LENs use human-understandable concept inputs and tailored learning objectives to produce predictions with FOL explanations, either for their own classifications or another classifier’s behavior.

    Source: §1 · p. 3, §3.1 · p. 8, §4.2 · p. 13

  • Results

    Across all considered tasks, µ and ReLU LENs are on the Pareto frontier and provide, on average, better accuracy–interpretability trade-offs; LEN explanations are less complex than BRL rules.

    Source: §6.3 · pp. 23–24

  • Takeaways & Limitations

    LENs can be configured as interpretable classifiers or explanatory modules, with FOL rules tailored to different objectives and scopes.

    Source: §1 · p. 3, §4.2 · p. 13, §7 · p. 27

    LENs require symbolic input and output spaces, which narrows their applications; computer-vision use may require additional concept annotations and attribute labels.

    Source: §7 · p. 27

Abstract

from arXiv · show

The large and still increasing popularity of deep learning clashes with a major limit of neural network architectures, that consists in their lack of capability in providing human-understandable motivations of their decisions. In situations in which the machine is expected to support the decision of human experts, providing a comprehensible explanation is a feature of crucial importance. The language used to communicate the explanations must be formal enough to be implementable in a machine and friendly enough to be understandable by a wide audience. In this paper, we propose a general approach to Explainable Artificial Intelligence in the case of neural architectures, showing how a mindful design of the networks leads to a family of interpretable deep learning models called Logic Explained Networks (LENs). LENs only require their inputs to be human-understandable predicates, and they provide explanations in terms of simple First-Order Logic (FOL) formulas involving such predicates. LENs are general enough to cover a large number of scenarios. Amongst them, we consider the case in which LENs are directly used as special classifiers with the capability of being explainable, or when they act as additional networks with the role of creating the conditions for making a black-box classifier explainable by FOL formulas. Despite supervised learning problems are mostly emphasized, we also show that LENs can learn and provide explanations in unsupervised learning settings. Experimental results on several datasets and tasks show that LENs may yield better classifications than established white-box models, such as decision trees and Bayesian rule lists, while providing more compact and meaningful explanations.

1 Introduction

The introduction motivates explanations that people can understand in consequential settings and presents Logic Explained Networks (LENs) as neural models that express predictions through First-Order Logic (FOL). It outlines LENs’ configurable uses and reports classification comparisons with established white-box models.

1 Introduction

  • LENs address the opacity of neural networks, whose use in safety-critical domains is constrained by the need for human-understandable explanations.The introduction also notes that explanations should serve human purposes, while their definition and quality remain contested.Source: §1 · pp. 1–2
  • Concept-based methods use human-understandable symbols, but few explain how a black box combines concepts or quantitatively assess explanation validity.The paper presents FOL as a formal language for explanations that can be combined, measured, rewritten, and simplified.Source: §1 · p. 2
  • LENs generalize neural methods for categorical learning by taking human-understandable concepts as inputs and producing predictions with FOL explanations.Parsimony criteria help keep their structure simple; LENs can classify data directly or explain another black-box classifier.Source: §1 · pp. 2–3
  • Users can interconnect LENs and customize explanations, including logic rules with different scopes, from individual samples to subsets of data.The framework also covers multiple use cases, including supervised classification and unsupervised learning.Source: §1 · pp. 3–4
  • Three preset LENs may generalize better in accuracy than decision trees on complex Boolean tasks.The introduction places this comparison in line with Tavares et al. [Tavares et al., 2020].Source: §1 · p. 4

2 Related work

The related-work section organizes XAI approaches by explanation type, model role, and scope. It contrasts feature-scoring and rule-based methods, then positions LENs among interpretable models and explanation methods.

2 Related work

  • XAI approaches can be compared by explanation result, role, and scope, though existing taxonomies overlap and use terminology with some ambiguity.The section surveys feature-scoring versus rule-based results, interpretable models versus explanation methods, and local versus global explanations.Source: §2 · p. 5
  • Feature-scoring methods summarize input features, whereas rule-based approaches use formal languages and can apply learned rules directly to the task.The section notes that feature scores may be of limited utility in decision support.Source: §2 · p. 5
  • LIME and SHAP provide local feature-scoring explanations, while LENs aim to express explanations as FOL formulas.Other examples include gradient-based saliency maps and methods that aggregate local explanations into global ones.Source: §2 · p. 6
  • LENs resemble DeepRED in decomposing neural-network behavior, but can explain any black-box model with human-interpretable input and output categories.LENs support FOL rules with different scopes and can serve as interpretable classifiers or surrogate models.Source: §2 · p. 6
  • Unlike standard interpretable-by-design methods, LENs use neural networks whose representation capacity may improve generalization on complex relationships.Their extracted explanations match their own classifier’s predictions.Source: §2 · p. 6

3 Logic Explained Networks

Logic Explained Networks (LENs) map human-understandable input concepts to output concepts and translate their predictions into First-Order Logic explanations. They support direct classification, explanations of black-box predictions, unsupervised clustering, and layered concept hierarchies.

3 Logic Explained Networks

  • A LEN maps human-understandable input concepts to output concepts, with each trained output translated into a First-Order Logic rule and formula.The framework supports explanations with local or more global coverage of the data.Source: §3 · p. 7

3.1 Computational pipelines

  • End-to-end LENs use interpretable features directly as concepts and can classify with explanations or explain a parallel black-box classifier.Features must be scaled to [0, 1] or Booleanized; discretization is one option for generic continuous features.Source: §3.1 · p. 8, Figure 4 · p. 9
  • Concept-bottleneck LENs use a black-box model to produce interpretable input concepts, then predict higher-level concepts or discover clusters.The prediction task is supervised; the clustering variant trains the LEN without output labels.Source: §3.1 · pp. 8–9, Figure 5 · p. 10
  • Cascading LENs map concepts through successive abstraction levels, providing explanations at multiple granularities and supporting human intervention at different levels.The experiments consider supervised cascading LENs as multiple interpretable classifications.Source: §3.1 · p. 9, Figure 6 · p. 11

3.2 Objectives

  • Interpretable classification trains LEN outputs on class labels while constraining them to produce coherent logic rules linking input concepts to each class.A class explanation can express a discovered relationship among selected input concepts.Source: §3.2 · p. 10
  • When explaining a black-box, a LEN is trained to mimic its predictions, using output coherence as a learning criterion that need not require supervision.The same types of FOL rules are used as for interpretable classification.Source: §3.2 · p. 11
  • Interpretable clustering uses unsupervised learning to identify co-occurrences of input concepts in subsets, with output activations representing cluster membership.The output concepts are not associated with known categories.Source: §3.2 · p. 11
  • Across objectives, extracted logic rules can be instantiated as FOL formulas for explanations of individual examples, groups, or the dataset more broadly.Example- and set-level explanations can also be applied recursively in cascading LENs.Source: §3.2 · p. 11

4 Methods

The methods explain LEN outputs by converting Booleanized concept inputs and predictions into logic formulas, then constrain learning and network structure to produce the desired explanations. They support local through global explanations, supervised and unsupervised objectives, and pruning for parsimony.

4.1 Extraction of logic explanations

  • Example explanations can be aggregated into set- or class-level formulas; repeated patterns reduce complexity, while frequent explanations can exclude noisy outliers.Source: §4.1 · pp. 12–13, Figure 7 · p. 12
  • Thresholded input concepts and LEN predictions form an empirical truth table for each output, which is converted into a DNF formula over selected concepts.Source: §4.1 · p. 12

4.2 Learning criteria

  • Learning criteria connect logic rules to outputs through supervised IF, Only IF, or IFF rules, black-box mimicry, or unsupervised mutual-information clustering.Source: §4.2 · pp. 13–15

4.3 Parsimony

  • Parsimony limits each output's dependence on input concepts; L1 regularization shrinks less-important weights, while readability may suffer with many concepts or large support.Source: §4.1 · p. 13, §4.3 · p. 15

4.3.1 Pruning strategies

  • Node-level pruning fixes each neuron's fan-in and can reduce classification performance; network-level pruning removes less-relevant input concepts, while example-level pruning selects active paths per sample.Source: §4.3.1 · pp. 15–16

5 Out-of-the-box LENs

The out-of-the-box LENs illustrate different accuracy–interpretability trade-offs through three network designs: ψ, µ, and ReLU. Their approaches range from interpretable neurons to network-level or example-level explanations.

5.1 ψ Network

  • ψ networks use sigmoid activations, strong L1 regularization, and node-level pruning to make each neuron interpretable, but limited learning capacity can impair accuracy and explanation quality.Rules from neurons are combined into final explanations; the number of hidden layers should remain small to avoid rule-merging issues.Source: §5.1 · p. 18

5.2 µ Network

  • µ networks prune input concepts across the network, retaining learning capacity close to an unconstrained network while supporting concise, high-quality explanations.Hidden neurons are not individually interpretable as in ψ networks.Source: §5.2 · pp. 18–19
  • Because µ pruning retains the same input concepts for every class, it can be limiting when classes depend on different concepts; separate binary µ networks offer an adaptation.Source: §5.2 · p. 18

5.3 ReLU network

  • ReLU networks prune each example’s affine instance without altering the network structure, preserving its capacity for strong classification performance while offering only mild explanation capacity.The pruning uses ReLU activations and mild L1 regularization.Source: §5.3 · p. 19
  • ReLU explanations are example-specific and may differ substantially even among samples in the same class, so they may not capture the class globally or be optimal.Source: §5.3 · p. 19

6 Benchmarking out-of-the-box LENs

Across varied classification and clustering tasks, LENs provide competitive classification and logic explanations, with μ networks offering a particularly attractive balance of accuracy and rule simplicity. The experiments also examine extraction speed, consistency, and using LEN-derived rules to reject adversarial examples.

6.1 Dataset and classification task details

  • The benchmark spans five categorical learning problems, including ICU outcomes, bird-species classification, democracy classification, and unsupervised MNIST odd/even clustering.Tasks use end-to-end, concept-bottleneck, or cascading LEN pipelines, depending on the data and objective.Source: §6.1 · pp. 20–21, §6.3 · p. 24

6.2 Metrics

  • Evaluation covers model and explanation accuracy, explanation complexity, fidelity, consistency, and rule-extraction time, generally averaged over 10-fold cross-validation.CUB uses five folds because BRL takes about three hours per fold.Source: §6.2 · pp. 21–22

6.3 Results and discussion

  • LENs generally outperform decision trees and often BRL in classification, while their extracted explanations are usually similarly accurate and less complex; μ networks offer the strongest overall balance.ReLU networks achieve the best classification accuracy, with μ networks close behind; BRL reaches the top explanation accuracy in some cases.Source: §6.3 · pp. 22–23, Table 4 · p. 22, Table 5 · p. 22
  • Tree and BRL explanations trivially achieve 100% fidelity, while ReLU explanations are often higher than 90%; LENs extract rules one to three orders of magnitude faster than BRL.BRL takes about 1 hour in two cases and over 1 day for CUB; decision trees are fastest overall.Source: Table 7 · p. 23, §6.3 · p. 24
  • Across tasks, μ and ReLU LENs lie on the Pareto frontier for average model and explanation error, yielding the better accuracy–interpretability trade-offs.Figure 10 plots the two errors as 100 minus model or explanation accuracy.Source: §6.3 · p. 24, Figure 10 · p. 24
  • Representative extracted rules show μ and ψ explanations are compact, generally shorter than tree rules and often shorter than BRL rules.Tree rules are omitted from Table 9 because they involve many terms.Source: §6.3 · p. 24, Table 9 · p. 25
  • In unsupervised MNIST odd/even clustering, all methods achieve high cluster accuracy, but only ψ consistently explains clusters using just the odd/even labels, with 100% consistency.The task uses no supervision, and LENs are not told that odd and even are mutually exclusive.Source: §6.3 · p. 24, Table 10 · p. 26

6.4 LENs as Adversarial Defense

  • On CUB adversarial attacks, LEN-derived rules significantly improve the attacked classifier’s accuracy; ReLU rules defend better but add computational burden through greater complexity.The experiment uses APGD-CE with perturbation upper bound ϵ = 0.5.Source: §6.4 · p. 26, Figure 11 · p. 26

7 Conclusions and Future Work

Logic Explained Networks (LENs) provide First-Order Logic explanations either for black-box classifiers or for interpretable classification. The paper reports a favorable trade-off against state-of-the-art white-box classifiers, while identifying symbolic inputs and outputs as a key limitation.

7 Conclusions and Future Work

  • LENs provide First-Order Logic explanations for black-box models or solve classification problems interpretably.The explanation type is connected to the learning criterion driving training, and the framework covers varied pipelines and user objectives.Source: §7 · pp. 26–27
  • Three out-of-the-box LEN models show a balanced trade-off among important properties and compare favorably with state-of-the-art white-box classifiers.Source: §7 · p. 27
  • First-Order Logic explanations require symbolic input and output spaces, narrowing applications to symbolic problems; computer vision may require additional annotations and attribute labels.The paper also notes limited efficiency and classification performance for fully interpretable ψ networks, and reduced efficiency of μ networks on multi-class or multi-label tasks.Source: §7 · p. 27
  • The authors identify dynamic interaction with people, more natural-language explanations, and higher-order or temporal logic as directions for expanding the framework.They also suggest exploring graph neural networks where they may better suit the classification problem.Source: §7 · p. 27
  • The framework is presented as a way to support safer AI adoption, aid bias identification, and make black boxes more robust to adversarial attacks.The authors also describe potential uses in reverse engineering, vulnerability discovery, system design, and scientific knowledge distillation.Source: §7 · p. 27

A Python APIs for LENs

The authors released the PyTorch, Explain! package to make LENs accessible, with documentation and low-level APIs for custom designs. Its XOR example demonstrates training and pruning a ψ network, generating a class explanation, and measuring its accuracy and complexity.

A Python APIs for LENs

  • PyTorch, Explain! is a Python package released to make LEN paradigms accessible, with extensive documentation on methods and low-level APIs.Source: A Python APIs for LENs · p. 31
  • The low-level APIs allow users to design custom LENs, as illustrated by the XOR example.Source: A Python APIs for LENs · pp. 31–32
  • The XOR example trains and prunes a ψ network, generates a first-order logic explanation for a target class, and computes explanation accuracy and complexity.Source: A Python APIs for LENs · p. 32

Related papers

Loading 2108.05149v1…