PaperlayerView exact revision on arXiv

Source-linked AI summary

NeuroRule: Making Black-Box Neural Networks Explainable through Rule-set Evolution

Tapaswini Kodavanti, Hormoz Shahrzad, Risto Miikkulainen

arXiv:2609.26841v2cs.NEcs.LG

TL;DR

Neural networks can deliver strong classification performance but often lack transparency, while existing explainability approaches may not describe whole-model behavior. NeuroRule evolves concise propositional-logic rule-sets to emulate neural networks; the paper reports that these rules can be more accurate than unconstrained alternatives, generalize well, and be reconstructed without the original training data.

  • Problem

    High-performing neural networks are often black boxes, while simpler interpretable models may lack performance and post-hoc explanations often cover individual predictions rather than whole-model behavior.

    Source: §1 · p. 1, §6 · p. 19

  • Method

    NeuroRule evolves propositional-logic rule-sets to emulate neural networks, adds conciseness as an objective, and can sample the network when original training data is unavailable.

    Source: §1 · p. 1

  • Results

    Conciseness-constrained rule-sets were often more accurate than unconstrained single-objective models, and could generalize better than the neural network, especially on out-of-distribution cases.

    Source: §6 · p. 17, §8 · p. 20

  • Takeaways & Limitations

    The paper concludes that neural networks can serve as portable knowledge stores whose behavior can be reconstructed as human-readable logic, even when original training data is unavailable.

    Source: §8 · p. 20

    Performance may be limited on noisy or heavily skewed datasets when the neural network provides an insufficiently smooth fitness signal for rule-set evolution.

    Source: §6 · p. 19

Abstract

from arXiv · show

High-capacity neural network models have achieved state-of-the-art performance across diverse classification tasks, yet they frequently operate as black-box models, lacking the transparency necessary for critical decision-making. Such opacity creates a persistent trade-off between performance and explainability. This paper proposes a solution to address this gap: the NeuroRule knowledge distillation framework that results in explainable rule-sets from neural network models. NeuroRule adapts the EVOTER rule-set evolution infrastructure to treat neural networks as targets for the evolution process, distilling their performance into concise sets of propositional logic expressions. There are three primary contributions: (1) an evolutionary method for distilling black-box neural network models into explicit rule-set models; (2) a method for making rule sets more explainable by including a conciseness objective to evolution; and (3) a demonstration that the distillation is viable even without access to the original neural network training data. The paper thus establishes that black-box neural network models can be made explainable and therefore useful in real-world applications where trustworthiness is paramount.

1 Introduction

NeuroRule distills black-box neural networks into explainable rule-sets, addressing the tension between predictive performance and transparency. It often outperforms EVOTER and sometimes the target network, while conciseness and neural-network sampling support simpler rules and distillation without original training data.

1 Introduction

  • High-capacity neural networks can perform strongly but remain opaque, while simpler models may be easier to understand and less performant.Post-hoc methods such as LIME and SHAP explain individual instances but often do not describe entire-model behavior.Source: §1 · p. 1
  • NeuroRule evolves explainable rule-sets against a neural network target, adds conciseness as an objective, and can sample the network when its original training data is unavailable.The framework builds on EVOTER.Source: §1 · p. 1
  • NeuroRule often outperforms EVOTER and sometimes the target neural network, particularly on out-of-distribution samples.The paper suggests neural-network samples may be more regularized and extracted rules may capture general principles.Source: §1 · p. 2
  • NeuroRule provides a pathway from black-box networks to human-readable logic for applications where explainability is critical.Source: §1 · p. 2

2 Related Work

Related work contrasts local post-hoc explanations with models designed to be interpretable. It presents EVOTER as a rule-set foundation whose use for neural-network emulation motivates adding a conciseness objective.

2.1 Post-hoc Explanation methods for Neural Networks

  • LIME fits a local linear surrogate using perturbed instances, whereas SHAP assigns feature importance through Shapley values.The paper notes that LIME explanations can vary with sampling.Source: §2.1 · p. 2
  • LIME and SHAP explain individual predictions but do not provide a comprehensive logical account of a model’s decision behavior or a standalone classifier.The paper distinguishes transparency as elements, interpretability as mechanisms, and explainability as understanding, while noting that usage overlaps.Source: §2.1 · p. 2, §2 · p. 2

2.2 EVOTER Framework

  • EVOTER represents behavior with linear lists of propositional-logic rules, supporting human-readable and editable models.Using EVOTER to emulate black-box neural networks exposes a large search space and an accuracy-complexity trade-off; straightforward application can yield overly complex rule-sets.Source: §2.2 · pp. 2–3

3 Methodology

NeuroRule uses EVOTER’s propositional-logic rule-sets and a specialized genetic algorithm to search for models that emulate neural-network behavior. Its representation combines conditional rules with a default action, while evolutionary operators modify conditions, rules, and whole rule-sets.

3.1 Rule-Set Model Framework

  • NeuroRule adopts EVOTER’s propositional-logic framework to represent model behavior and support single- and multi-objective optimization.Source: §3.1 · p. 3

3.1.1 Rule-Set Representation.

  • Rule-sets use a linear list of IF-THEN rules whose antecedents conjoin conditions and whose separate rules are joined by disjunction.The grammar includes feature comparisons, exponentiation, and time lags, but this paper omits time lags to simplify classification search.Source: §3.1.1 · p. 3, Figure 1 · p. 4
  • Triggered rules propose class labels with coefficients, and the final prediction is the label with the highest cumulative coefficient; a default rule handles cases where no other rule applies.The default captures the most common class, allowing rules to evolve exceptions.Source: §3.1.1 · p. 4, Figure 2 · p. 4

3.1.2 Evolutionary Optimization.

  • A specialized genetic algorithm searches a variable-length, discrete space using crossover and mutation.Crossover includes single-point, uniform, and combine-rules variants; mutation operates at condition, rule, and rule-set levels.Source: §3.1.2 · p. 5
  • Mutations can change thresholds or operators, alter a rule’s conditions, or add, remove, and reorder rules or change the default; contradictory or redundant mutations are discarded.Combine-rules crossover merges conditions and rules sharing an action, while uniform crossover samples rules from parent sets.Source: §3.1.2 · p. 5

3.2 Fitness Function Design

NeuroRule’s fitness design balances fidelity to neural-network predictions with rule-set conciseness. It uses a Pareto-based optimizer to retain trade-offs between accuracy and complexity.

3.2.1 Accuracy Objective.

  • The accuracy objective measures the proportion of classifications in which a candidate rule-set matches the target neural network.The search maximizes accuracy in predicting the network’s outputs over a set of data points.Source: §3.2.1 · p. 5

3.2.2 Conciseness Objective.

  • The conciseness objective minimizes rule-set complexity to promote readability, using the total number of atomic conditions as a proxy for explainability.Fewer rules, smaller rule structures, and fewer literals reduce the cognitive overhead of understanding model behavior.Source: §3.2.2 · pp. 5–6

3.2.3 Multiobjective optimization.

  • NSGA-II optimizes accuracy and conciseness by maintaining a Pareto front of non-dominated rule-sets.Non-dominated sorting ranks solutions across objectives, while crowding distance favors less populated regions and helps preserve a spread of complexity trade-offs.Source: §3.2.3 · p. 6

3.3 Knowledge Distillation Pipeline

NeuroRule distills a trained neural network by evolving rule-sets to reproduce its predictions, with the network guiding fitness evaluation. A multi-objective version adds conciseness alongside performance.

3.3 Knowledge Distillation Pipeline

  • The trained neural network acts as the teacher, guiding evolutionary search by its behavior rather than the original training data.Rule-sets are evaluated against NN-generated predictions instead of the original labels.Source: §3.3 · p. 6
  • The pipeline first trains the NN on dataset D to predict its labels, then integrates it into the evolutionary fitness function.Each generation compares rule-set predictions with the NN’s predictions, optimizing reconstruction of NN behavior rather than fit to original labels.Source: §3.3 · p. 6
  • The performance-and-explainability setting uses a multi-objective algorithm to optimize both objectives.The process is illustrated in Figure 4.Source: §3.3 · p. 6

4 Experimental Setup

The experiments evaluate NeuroRule on clinical classification data, test in-distribution and out-of-distribution generalization, and examine distillation without original training data. They compare network-guided rule evolution with EVOTER and assess performance and conciseness.

4.1.1 Clinical Datasets.

  • The study evaluates NeuroRule on three medical classification datasets from the UC Irvine collection.The setup also describes the data partitions and a synthetic dataset generated from NN behavior.Source: §4.1.1 · p. 7, §4.1 · p. 7

4.1.2 ID/OOD Partitioning of Data.

  • Training uses middle-distribution samples as in-distribution data and tail samples as out-of-distribution data.The NN is trained with ID portions comprising the middle 80%, 90%, or 95%; the setup tests scaling with training-data availability and extrapolation farther from training data.Source: §4.1.2 · pp. 7–8

4.1.3 Creating Synthetic Data.

  • When original training data are unavailable, random feature-space inputs are labeled by the NN to create the synthetic dataset used for rule-set evolution.Sampled feature values are constrained to their original domains; this random sampling of the decision manifold can cover situations underrepresented in the original data, producing a slightly different distillation result.Source: §4.1.3 · p. 8

4.2.1 NN Parameters.

  • The target NN is a funnel-shaped multilayer perceptron, with decreasing hidden-layer sizes from 32 to 16 and a final Softmax output discretized to binary labels.The labels generated by the NN replace original labels in the distillation fitness dataset.Source: Table 1 · p. 9, §4.2.1 · p. 9

4.2.2 Rule-set Evolution Parameters.

  • Rule-set evolution uses mutation probability 0.7 and novelty selection multiplier 1.5 to encourage broad exploration of the logical search space.The NN-guided fitness function is described as providing a stable environment for maintaining high mutation rates without losing prior evolutionary progress.Source: §4.2.2 · p. 9

4.2.3 EVOTER Baseline Comparison.

  • EVOTER evolves rule-sets directly from original data, providing a baseline against NeuroRule’s distillation from the NN’s smoothed decision manifold.The comparison is intended to isolate the impact of network guidance.Source: §4.2.3 · p. 9

4.2.4 Single-Objective vs. Multi-Objective Optimization.

  • Single-objective selection optimizes accuracy alone, whereas NSGA-II jointly optimizes accuracy and conciseness.The multi-objective goal is to find solutions as performant as single-objective solutions but more concise and explainable.Source: §4.2.4 · pp. 9–10

4.3 Metrics

  • Evaluation measures ID accuracy, OOD accuracy, and rule-set conciseness as total Condition Count.OOD accuracy is the primary measure of generalization; lower condition count indicates a more explainable model.Source: §4.3 · p. 10

5 Results

Across the experiments, NeuroRule rule-sets often outperform EVOTER and, on out-of-distribution data, outperform neural networks; adding conciseness reduces model complexity without sacrificing accuracy. Synthetic-data distillation remains feasible, though its rule-sets are less accurate on original samples.

5.1.1 In-Distribution Performance.

  • In most cases, NeuroRule accuracy plateaued slightly below the neural network, but NeuroRule significantly improved on EVOTER in most cases.The authors suggest concise rule-sets may miss training-data nuances, while neural-network targets can smooth noisy labels.Source: §5.1.1 · pp. 10–11

5.1.2 Out-of-Distribution Performance.

  • On out-of-distribution data, NeuroRule performed best and EVOTER also outperformed neural networks across the reported clinical benchmarks.The table reports averages over 10 runs; the authors suggest rule-sets extrapolate by capturing underlying data principles.Source: Table 4 · p. 11, §5.1.2 · p. 11

5.2.1 Effect on Explainability.

  • Adding a conciseness objective produced smaller rule-sets for both EVOTER and NeuroRule, with NeuroRule models consistently having lower condition counts.The authors note that these concise models are easier to verify with domain experts.Source: §5.2.1 · p. 13, Table 5 · p. 13

5.2.3 Accuracy vs. Conciseness tradeoff.

  • Multi-objective solutions performed more consistently, and many of the highest-accuracy solutions were also among the most concise.The conciseness constraint narrowed search variation; Figure 7 plots accuracy against conciseness and shows multi-objective solutions concentrated at lower complexity.Source: §5.2.3 · pp. 13–14, Figure 7 · p. 16

5.2.2 Effect on Performance.

  • Multi-objective evolution often improved accuracy rather than reducing it, with rule-sets generally performing better than accuracy-only counterparts and close to neural-network accuracy.The reported comparison uses averages across 10 experimental runs.Source: §5.2.2 · p. 13, Table 6 · p. 14

5.3 Experiment 3: Distilling with Synthetic Data

  • Synthetic-data rule-sets had lower accuracy on original samples than rule-sets evolved from original data, yet still achieved competent accuracy and demonstrated feasibility when training data are unavailable.The authors attribute the accuracy decrease to matching potentially inaccurate neural-network boundaries in sparsely sampled regions.Source: §5.3 · pp. 14–15

6 Discussion

The discussion argues that concise rule sets can challenge the performance–explainability trade-off, including through better generalization, while identifying data and search conditions that constrain NeuroRule.

The Performance vs. Explainability Trade-off

  • Conciseness-constrained models were often more accurate than unconstrained models, showing that greater explainability need not reduce performance.The discussion also says NN behavior can be distilled into minimal logical expressions without significant information loss.Source: §6 · p. 17

Symbolic Regularization through Conciseness Constraint

  • Penalizing rule conditions can prevent fitting training-set outliers and may improve generalization in noisy environments.Without the constraint, complex rule sets captured training noise; the multi-objective search instead favors dominant conditions.Source: §6 · p. 17

Distributional Invariance and Extrapolation

  • Concise rule sets may retain predictive power under distribution shifts by avoiding spurious correlations tied to the training distribution.The discussion contrasts rule sets as high-bias extrapolators with neural networks as high-variance interpolators.Source: §6 · p. 17

Synthetic-data distillation

  • NeuroRule evolved with synthetic samples had lower accuracy than versions trained with original data, but remained viable for producing explanations.The paper describes the network as a portable domain-knowledge source when original training data are unavailable.Source: Table 7 · p. 17, Figure 8 · p. 18

The NN as a Regularizer

  • The neural network provides a smoothed decision manifold against which rules evolve, partially shielding the search from data noise.The discussion reports that this approach performs better than EVOTER evolved directly with data.Source: §6 · pp. 18–19

Limitations

  • Distillation may converge suboptimally when feature composition, high-frequency noise, or skewed distributions make the neural network's fitness signal unsuitable.The discussion also notes that evolutionary search is stochastic; larger populations and more sophisticated mutation operators are proposed for more consistent convergence.Source: §6 · p. 19

7 Future Work

Future work proposes broadening NeuroRule's output types and input domains, examining neural-network architecture sensitivity, and making rule-set interventions easier for domain experts.

NN architecture sensitivity

  • The authors propose testing whether NeuroRule's results depend on neural-network architecture, including network depth and activation functions.The rule-set's accuracy depends directly on neural-network accuracy, while architecture sensitivity remains possible.Source: §7 · p. 19

Rule-set probability outputs

  • The framework could be extended to probabilistic or distributional rule outputs for applications involving non-deterministic actions.The authors give robotics and control-based environments as examples.Source: §7 · p. 19

Modeling beyond feature comparisons

  • Extensions to 2D/3D inputs and language-based comparisons could support distillation of CNNs and sequence-based networks such as RNNs.Source: §7 · p. 19

Explainability for vector embeddings

  • Applying NeuroRule to vector embeddings could help connect embedding values with human-readable values and provide insights into large language models.Source: §7 · p. 19

Interactive framework

  • An interactive framework could let domain experts monitor, validate, and edit NeuroRule's explicit rule sets.The proposed human-in-the-loop intervention framework is intended to broaden applications beyond research settings.Source: §7 · p. 19

8 Conclusion

The conclusion presents NeuroRule as a way to express neural-network behavior in human-readable rule sets, with accuracy, conciseness, and synthetic-data distillation as central findings.

Conclusion

  • NeuroRule distills neural-network behavior into transparent rule sets, and the authors argue this can preserve performance without sacrificing explainability.Source: §8 · p. 20
  • Optimizing rules to emulate the neural network can improve performance, while a conciseness objective reduces rule complexity and may improve out-of-distribution generalization.The conclusion says concise rule sets may generalize better than the neural network, especially in OOD cases.Source: §8 · p. 20
  • NeuroRule can reconstruct neural-network behavior without the original training data, and the authors identify safety-critical industries as a potential application area.Source: §8 · p. 20

A Experiment 1 Learning Curves for the Breast Cancer and Heart Disease Datasets

The appendix compares EVOTER and NeuroRule learning curves with neural-network accuracy across ID/OOD splits, reporting dataset-specific patterns and multi-objective results.

Experiment 1 Learning Curves

  • On Breast Cancer, NeuroRule and EVOTER learning-curve results were similar to Diabetes, while EVOTER accuracy showed more variance, likely due to the smaller dataset.Source: Figure 9 · p. 22
  • Rule-set approaches were more accurate than the neural network in five of six Heart Disease cases, including the 80% and 90% ID cases.Source: Figure 10 · p. 23

Experiment 2 Learning Curves

  • For multi-objective optimization on Breast Cancer and Heart Disease, the reported results show significant improvement over single-objective evolution and less variance, similar to Diabetes.For Breast Cancer, ID results begin as vertical lines with limited data and become more scattered as training data increase.Source: Figure 11 · p. 25, Figure 12 · p. 26

Experiment 2 Accuracy vs. Conciseness

C Experiment 3 Learning Curves for Breast Cancer and Heart Disease Datasets

The learning curves compare EVOTER and NeuroRule with neural-network accuracy across ID/OOD splits for Breast Cancer and Heart Disease when NeuroRule uses synthetic data only. Accuracy is lower but remains viable, with results similar to those on Diabetes.

C Experiment 3 Learning Curves for Breast Cancer and Heart Disease Datasets

  • With synthetic data only, NeuroRule achieves lower but still viable accuracy across ID/OOD splits on Breast Cancer, compared with neural-network accuracy.Figure 15 shows learning curves for EVOTER and NeuroRule alongside neural-network accuracy.Source: Figure 15 · p. 29
  • The Breast Cancer and Heart Disease results are similar to those on Diabetes.The learning curves for Breast Cancer and Heart Disease are shown in Figures 15 and 16.Source: Figure 15 · p. 29, C Experiment 3 Learning Curves for… · p. 29, Figure 16 · p. 30
  • With synthetic data only, NeuroRule achieves lower but still viable accuracy across ID/OOD splits on Heart Disease, compared with neural-network accuracy.Figure 16 shows learning curves for EVOTER and NeuroRule alongside neural-network accuracy.Source: Figure 16 · p. 30
Loading 2609.26841v2…