PaperlayerView exact revision on arXiv

Source-linked AI summary

Interpretable Neural-Symbolic Concept Reasoning

Pietro Barbiero, Gabriele Ciravegna, Francesco Giannini, Mateo Espinosa Zarlenga, Lucie Charlotte Magister, Alberto Tonda, Pietro Lio', Frederic Precioso, Mateja Jamnik, Giuseppe Marra

arXiv:2304.14068v2cs.AIcs.LGcs.NEstat.ML

TL;DR

Deep learning’s opaque decisions and concept embeddings’ unclear semantics limit trust in concept-based models. DCR builds fuzzy rules from concept embeddings and executes them on meaningful concept truth degrees; on challenging datasets, it improves ROC-AUC by up to ~25% over interpretable baselines.

  • Problem

    Concept-based models either struggle to solve real-world tasks using concept truth-values alone or use embeddings whose dimensions lack clear semantic meaning, limiting interpretability.

    Source: §1 · p. 1, §5 · p. 9

  • Method

    DCR uses learnable differentiable modules to build fuzzy rules from concept embeddings, then executes those rules on semantically meaningful concept truth degrees to produce predictions.

    Source: §1 · p. 1

  • Results

    DCR improves ROC-AUC by up to ~25% over interpretable baselines on challenging datasets where concept embeddings contain more information than concept truth degrees.

    Source: §4.3 · p. 6

  • Takeaways & Limitations

    DCR discovers meaningful rules matching known ground truths without concept supervision, and those rules can guide counterfactual-example generation.

    Source: §1 · p. 1

    DCR requires concept embeddings, assuming concept-based datasets or high-quality concept-discovery methods; its global rules may not exactly reflect the model’s reasoning.

    Source: §5 · p. 8

Abstract

from arXiv · show

Deep learning methods are highly accurate, yet their opaque decision process prevents them from earning full human trust. Concept-based models aim to address this issue by learning tasks based on a set of human-understandable concepts. However, state-of-the-art concept-based models rely on high-dimensional concept embedding representations which lack a clear semantic meaning, thus questioning the interpretability of their decision process. To overcome this limitation, we propose the Deep Concept Reasoner (DCR), the first interpretable concept-based model that builds upon concept embeddings. In DCR, neural networks do not make task predictions directly, but they build syntactic rule structures using concept embeddings. DCR then executes these rules on meaningful concept truth degrees to provide a final interpretable and semantically-consistent prediction in a differentiable manner. Our experiments show that DCR: (i) improves up to +25% w.r.t. state-of-the-art interpretable concept-based models on challenging benchmarks (ii) discovers meaningful logic rules matching known ground truths even in the absence of concept supervision during training, and (iii), facilitates the generation of counterfactual examples providing the learnt rules as guidance.

1. Introduction

Concept-based models use human-understandable concepts to support interpretability, but concept embeddings lack clear semantics. DCR builds fuzzy rules from embeddings and evaluates them on meaningful concept truth degrees.

1. Introduction

  • Concept-based models can aid understanding, but embedding dimensions lack clear semantics, weakening interpretability.The paper notes that embedding-based models trade some interpretability for model capacity.Source: §1 · p. 1
  • DCR builds fuzzy rules with learnable modules on concept embeddings, then evaluates them on concept truth degrees for interpretable predictions.The introduction presents DCR as the first interpretable concept-based model built on concept embeddings.Source: §1 · p. 1
  • DCR discovers meaningful logic rules matching known ground truths without concept supervision and facilitates rule-guided counterfactual generation.Source: §1 · p. 1
  • DCR outperforms interpretable concept-based models on the Dot dataset, and the experiments report better task accuracy than state-of-the-art interpretable models.Figure 1(b) shows the Dot comparison.Source: §1 · p. 1, Figure 1 · p. 2

2. Preliminaries

Concept-based models map concepts to task labels, optionally through a concept encoder. Their representations range from truth degrees to embeddings, while fuzzy logic supplies differentiable operators for rules.

2. Preliminaries

  • A concept-based model maps a concept space to task labels; a concept encoder can first map hard-to-interpret input features into that space.Training may use task labels and, when available, ground-truth concept annotations.Source: §2 · p. 2
  • Concept truth degrees are values in [0, 1], while embeddings add high-dimensional representations that can improve accuracy but weaken interpretability.Embedding dimensions lack clear semantics.Source: §2 · p. 2
  • Fuzzy logic extends Boolean truth values to degrees in [0, 1] and replaces Boolean connectives with differentiable real-valued operators.The passage describes t-norm conjunction, t-conorm disjunction, and negation as 1 − x.Source: §2 · p. 2

3. Deep Concept Reasoning

DCR uses concept embeddings to construct logic rules, then predicts by evaluating those rules on concept truth degrees. Its rule syntax, differentiable generation, parsimony controls, and explanation procedures make this design operational.

3. Deep Concept Reasoning

  • DCR uses embeddings only to compute logic rules, then evaluates rules on concept truth values for semantically clear predictions.The system is differentiable and can be trained alone on concept databases or end-to-end with differentiable concept encoders.Source: §3 · p. 2

3.1. Rule syntax

  • Each class rule is a conjunction of concept literals, with each concept assigned a positive or negative role and marked relevant or irrelevant.The banana example illustrates that a concept such as “soft” can be irrelevant while “not round” and “yellow” are relevant.Source: §3.1 · p. 3

3.2. Rule generation and execution

  • Neural modules φ and ψ infer concept roles and relevance from embeddings; DCR combines the resulting literals and executes the rule on concept truth degrees.The modules are differentiable and do not depend on the number of input concepts, allowing use with a different available concept set at test time without retraining.Source: §3.2 · pp. 3–4

3.3. Rule parsimony and fuzzy semantics

  • A rescaled log-softmax activation encourages rule parsimony, while the Gӧdel t-norm supplies differentiable fuzzy semantics for the logic operations.The parameter τ lets users bias relevance scores toward more or fewer selected concepts.Source: §3.3 · p. 4

3.4. Global and counterfactual explanations

  • DCR aggregates Booleanized training rules with a global disjunction for class-level explanations and uses ranked relevant concepts to guide counterfactuals.For counterfactuals, it inverts concept truth values in relevance order until the model prediction changes.Source: §3.4 · pp. 4–5

4. Experiments

The experiments test DCR’s generalization and interpretability across six datasets spanning tabular, image, and graph data. Results cover benchmark accuracy, rule discovery, explanation stability, counterfactuals, and diagnosis of prediction errors.

4.1. Research questions

  • The experiments ask whether DCR generalizes on unseen samples and with unsupervised concepts, and whether it discovers meaningful, stable rules that support counterfactuals.Source: §4.1 · p. 5

4.2. Experimental setup

  • The evaluation spans six datasets across tabular, image, and graph data, including synthetic rule tasks, MNIST-Addition, CelebA concept shift, and unsupervised Mutagenicity concepts.The study assesses ROC AUC, rule correctness, explanation sensitivity, and the number of concept perturbations needed for counterfactuals.Source: §4.2 · pp. 5–6

4.3. Task generalization

  • Up to ~25% improvement in ROC-AUC over interpretable baselines appears on challenging tasks where concept embeddings hold more information than truth degrees.On CelebA concept shift, interpretable models behave almost randomly, while DCR generalizes well by using information about correlations between train and test concepts.Source: §4.3 · p. 6, Figure 3 · p. 6
  • DCR achieves accuracy within 1% of the best neural-symbolic baseline on MNIST-Addition while discovering rules directly from data without concept supervision.The baselines are trained using ground-truth rules; DCR learns concepts without supervision and learns its rules from scratch.Source: §4.3 · p. 6, Table 1 · p. 7

4.4. Interpretability

  • DCR rules align with ground-truth rules on XOR, Trigonometry, and MNIST-Addition, and Booleanised-rule error rates quantify disagreement with the model’s fuzzy rules.On Mutagenicity, where ground-truth rules are unavailable, many learned rules include functional groups such as phenols and dimethylamines associated with harmful effects.Source: §4.4 · pp. 6–7, Table 2 · p. 7
  • DCR explanations are stable under small input perturbations, particularly compared with LIME and ReluNet; logistic regression and decision-tree explanations are fixed rules.Figure 4 compares explanation sensitivity as the perturbation radius increases; logistic regression and decision trees have trivially zero sensitivity because they learn fixed rules.Source: §4.4 · p. 7, Figure 4 · p. 8
  • DCR rules can explain task-level misclassifications by identifying wrongly predicted concepts or a suboptimal selection of relevant concepts.On XOR, DCR identifies a mislabeled sample: its learned rule is correct even though the ground-truth label is incorrect.Source: §4.4 · p. 7, Table 3 · p. 8
  • DCR rules guide counterfactual search by ranking concepts for perturbation, and prediction confidence drops quickly when the most relevant concepts are changed.DCR has the lowest area under the confidence curves in three of four datasets, indicating fewer concept perturbations are needed to find counterfactuals.Source: §4.4 · p. 8, Figure 5 · p. 8

5. Key findings & significance

DCR combines concept embeddings with explicit logic rules to retain semantic interpretability while supporting accurate predictions. Its strongest fit is challenging tasks where users need interpretability and existing interpretable models fall short, though global behavior and input requirements remain limitations.

Key findings & significance

  • DCR's global behavior may not be directly interpretable, its rules can grow complex when many concepts distinguish tasks, and it requires concept embeddings.The last requirement assumes concept-based datasets or high-quality concept-discovery methods; rule complexity is said to arise rarely in current benchmarks and most real-world cases.Source: §5 · p. 8
  • Concept-based explanations use human-interpretable concepts to make decisions easier to trace, particularly when input features such as pixels are hard to reason about.Prior work also identifies brittleness in post-hoc explainability methods as a drawback.Source: §5 · p. 8
  • Unlike neural-symbolic approaches that exploit supplied logic knowledge, DCR focuses on learning logical knowledge and exposes its decision process through executable rules.Other approaches discussed still rely on an uninterpretable neural component for decision-making.Source: §5 · p. 9
  • DCR is most relevant when interpretability is crucial and existing interpretable models fail; simpler interpretable models may suffice for simpler tasks.The paper contrasts this with black-box models when interpretability is not a firm requirement, and notes that methods such as DeepProbLog may be more stable when the full rule set is known.Source: §5 · p. 9
  • DCR builds logic rules from concept embeddings and executes them on concept truth degrees, keeping the prediction decision process explicit.The conclusion describes a weighted logic rule combining neural and symbolic algorithms in an end-to-end differentiable system.Source: §5 · p. 9

A. Datasets & Experimental Setup

The experiments span synthetic concept tasks and real-world image, graph, and digit-pair benchmarks. The datasets vary in how concepts and task labels are constructed, including settings with shifted or unavailable concept annotations.

A. Datasets & Experimental Setup

  • The XOR dataset uses two binary concepts thresholded from uniform samples in the unit square, with the task label defined as their XOR.The cited inspiration is the XOR problem used to illustrate Perceptron limitations.Source: A. Datasets & Experimental Setup · p. 13
  • The Trigonometric dataset transforms three latent normal variables into seven features and labels samples by whether the first two latent variables sum to a positive value.Its three binary concepts encode the signs of the latent variables, making the task Boolean-undecidable from those concepts.Source: A. Datasets & Experimental Setup · p. 13
  • The Vector dataset constructs features from latent vectors and labels whether two vectors point in the same direction, testing embedding representations against fuzzy concepts.Its concepts encode whether each latent vector aligns with fixed reference vectors.Source: A. Datasets & Experimental Setup · p. 13
  • MNIST-Addition pairs two digit images and labels each pair with their sum; Mutagenicity classifies molecular graphs as mutagenic or non-mutagenic.MNIST-Addition contains 30,000 labelled pairs, while Mutagenicity contains 4,337 graphs.Source: A. Datasets & Experimental Setup · p. 13
  • The CelebA task shifts from three frequent training attributes to a different, correlated set of three test attributes, using zero-padding for baselines.The training and test concept sets differ but their embeddings remain correlated; DCR handles the shift without modification.Source: A. Datasets & Experimental Setup · p. 13

B. Training details

Training uses a shared fuzzy-logic setup for DCR, with different concept-embedding generators across synthetic, image, and graph settings. Training schedules and optimization choices also vary by task.

B.1. Deep Concept Reasoner

  • DCR uses Godel t-norm semantics and two-layer MLPs for its neural modules; training runs for 3,000 epochs on synthetic and CelebA tasks and 7,000 on Mutagenicity.The temperature is 100 for both schedules.Source: §B.1 · p. 13

B.2. Concept Embedding Generators

  • Concept embeddings come from MLPs on synthetic data, a pretrained ResNet-34 on CelebA, and a GNN plus GCExplainer on Mutagenicity.Embedding sizes are 128 activations for synthetic tasks and 16 for CelebA. For Mutagenicity, GCExplainer extracts 30 concepts by clustering graph-node embeddings; the GNN uses four graph-convolution layers and is trained with an 80:20 train-test split.Source: §B.2 · p. 14
  • Synthetic tasks use 3,000 samples, a 70%-10%-20% split, concept-loss weight α = 1, 500 epochs, batch size 256, and Adam at 10^-2.CelebA uses concept-loss weight α = 1, weighted cross-entropy for concept prediction, 200 epochs, batch size 512, and SGD with 0.9 momentum at 5 × 10^-3; all models use weight decay and validation-based learning-rate reduction and early stopping.Source: §B.2 · p. 14

B.3. Hyperparameter search for benchmark classifiers

Benchmark classifier settings are selected with an internal three-fold cross-validation grid search. The search varies decision-tree, logistic-regression, and XGBoost parameters.

B.3. Hyperparameter search for benchmark classifiers

  • Benchmark classifiers are tuned with a grid search using internal three-fold cross-validation.The parameter grid includes settings for decision trees, logistic regression, and XGBoost.Source: §B.3 · p. 14
  • The decision-tree grid varies maximum depth, minimum samples to split, and minimum samples per leaf.The listed values are max depth [2, 4, 10, all leaves pure], min samples split [2, 4, 10], and min samples leaf [1, 2, 5, 10].Source: §B.3 · p. 14
  • The logistic-regression grid tests l1, l2, and elasticnet penalties, while the XGBoost grid tests tree, linear, and dart boosters.Source: §B.3 · p. 14

C. Mutagenicity: Extracted Concepts

The Mutagenicity analysis visualizes GCExplainer concepts through graph neighborhoods and finds recurring molecular structures associated with different predictions. A separate figure covers a softmax-temperature ablation on pretrained CUB concept embeddings.

C. Mutagenicity: Extracted Concepts

D. Softmax temperature effect on relevant concepts

  • Figure 11 presents a softmax-temperature ablation using pretrained concept embeddings from CUB.Source: Figure 11 · p. 21

E. Number of concepts effect on training and test time

On pretrained CUB concept embeddings, DCR's computational time rises linearly for small numbers of training concepts and then becomes almost constant.

  • Training time increases linearly for small concept counts, then becomes almost constant as the number of training concepts grows.The experiment trains DCR with 10, 50, 100, and 150 randomly selected concepts across five initialization seeds.Source: E. Number of concepts effect on… · p. 21, Figure 12 · p. 21

F. Sensitivity analysis

DCR's local explanations are typically about as stable as those of existing interpretable models, and its explanations yield the lowest counterfactual-curve values on three of four datasets.

F. Sensitivity analysis

  • DCR sensitivity is typically close to existing interpretable models, while LIME and ReluNet have higher sensitivity under non-significant perturbations.The sensitivity AUC measures explanation stability as perturbation radius increases; lower is better.Source: F. Sensitivity analysis · pp. 21–22, Table 4 · p. 22

G. Counterfactual explanations

H. MNIST addition experiment

In MNIST addition, DCR learns global rules that largely match addition semantics without direct concept labels; the paper also compares rule complexity and reports implementation resources.

H. MNIST addition experiment

  • MNIST addition pairs two images and labels their digit sum, while each image is individually unlabeled; the mapping uses 20 image-class concepts and 19 sum tasks.Source: H. MNIST addition experiment · p. 23
  • Without direct concept supervision, the network can use the full categorical distribution as an embedding for digit identity; the proposed solution supports unsupervised or distant concept learning.Source: H. MNIST addition experiment · pp. 23–24
  • DCR uses Gumbel-softmax sampling to encourage crisp concept decisions and adds a task predictor trained from concept probabilities.The added loss encourages that predictor to map the concept probabilities to the task label, rather than relying only on embeddings.Source: H. MNIST addition experiment · p. 23
  • The learned MNIST-addition rules match the task semantics except for one identifiable rule with a count of 1.Table 7 lists global rules learned from 10,000 training examples; network concept IDs were mapped to human digits after training.Source: Table 7 · p. 24

I. Complexity of logic rules

  • Rule complexity is the average size of learned rules; on most datasets it matches decision-tree complexity while DCR performs better, but Mutagenicity has a performance–complexity tradeoff.Source: §I · p. 25, Table 8 · p. 25
  • DCR's rules are per-sample, and the authors note that global rules could become more complex, especially when multiple concept combinations yield the same prediction.Source: §I · p. 25

J. Code, Licences, Resources

  • Experiments used Python 3.7, PyTorch 1.11, and Scikit-learn; the authors report an estimated 240 GPU-hours on a machine with two Quadro RTX 8000 GPUs.They state that experiment-recreation code will be released in a public MIT-licensed repository.Source: J. Code, Licences, Resources · p. 25
Loading 2304.14068v2…