Source-linked AI summary
Local Rule-Based Explanations of Black Box Decision Systems
Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Dino Pedreschi, Franco Turini, Fosca Giannotti
TL;DR
Black-box decision systems can produce opaque, potentially biased outcomes, creating a need for explanations of individual decisions. LORE generates a local neighborhood with a genetic algorithm, learns an interpretable decision tree, and extracts decision and counterfactual rules; experiments report that it outperforms existing methods and baselines in explanation quality and black-box mimicry.
Problem
Opaque black-box decisions lack explanations of why a specific outcome was assigned, limiting trust and adoption in sensitive contexts.
Method
LORE generates a balanced synthetic neighborhood with a genetic algorithm, learns a local decision tree, and extracts decision and counterfactual rules.
Results
LORE outperforms existing methods and baselines in explanation quality and accuracy in mimicking the black box.
Takeaways & Limitations
LORE provides black-box-agnostic local explanations that state why an outcome occurred and what feature changes could produce a different outcome.
Takeaways & Limitations
The paper identifies human comprehensibility evaluation and broader extensions as future work, and assumes users can understand elementary logic rules.
Abstract
from arXiv · showhide
The recent years have witnessed the rise of accurate but obscure decision systems which hide the logic of their internal decision processes to the users. The lack of explanations for the decisions of black box systems is a key ethical issue, and a limitation to the adoption of machine learning components in socially sensitive and safety-critical contexts. %Therefore, we need explanations that reveals the reasons why a predictor takes a certain decision. In this paper we focus on the problem of black box outcome explanation, i.e., explaining the reasons of the decision taken on a specific instance. We propose LORE, an agnostic method able to provide interpretable and faithful explanations. LORE first leans a local interpretable predictor on a synthetic neighborhood generated by a genetic algorithm. Then it derives from the logic of the local interpretable predictor a meaningful explanation consisting of: a decision rule, which explains the reasons of the decision; and a set of counterfactual rules, suggesting the changes in the instance's features that lead to a different outcome. Wide experiments show that LORE outperforms existing methods and baselines both in the quality of explanations and in the accuracy in mimicking the black box.
1 INTRODUCTION
LORE addresses the need to explain opaque, instance-specific decisions from black-box systems. It combines genetic-algorithm neighborhood generation with local rule-based explanations that include both decision and counterfactual rules.
- Motivation: Opaque machine-learning decision systems can inherit biased or misleading patterns from training data, producing discriminatory or wrong decisions.The lack of explanations for such decisions limits social acceptance and trust in sensitive contexts.
- Problem: The paper targets explanations of a specific black-box outcome rather than general descriptions of the entire system.The intended explanation should clarify why that decision was made for the affected user.
- Method: LORE generates a balanced neighborhood around an instance with a genetic algorithm, learns a local decision tree, and extracts an explanation from it.The method operates on black-box input-output behavior and does not require access to the system’s internals or training set.
- Explanation: Each LORE explanation contains a decision rule identifying why the instance received its outcome and counterfactual rules describing changes that would reverse it.This provides both evidence for the observed decision and alternatives for a different outcome.
- Evaluation: LORE is evaluated through extensive quantitative and qualitative experiments assessing explanation quality and fidelity to the black box.The paper reports that its approach outperforms existing methods and baselines on these dimensions.
2 RELATED WORK
Related work distinguishes global and local explanations, as well as model-dependent and black-box-agnostic approaches. LORE is positioned as combining black-box agnosticism with both decision rules and counterfactual rules for local explanations.
- Explanation scope: Many explanation methods build interpretable classifiers that mimic a black box globally, while others explain the local behavior around a specific instance.Existing approaches include model-dependent methods and a smaller number of agnostic methods.
- Local methods: LIME, Anchors, and related methods construct local explanations, but their neighborhood generation process is identified as a crucial weakness.The paper contrasts its focused genetic-algorithm exploration with the random neighborhood generation used by LIME-derived approaches.
- Counterfactual explanations: Counterfactual approaches can return a nearby instance with a different black-box outcome by solving an optimization problem.LORE instead expresses counterfactuals as logic rules characterizing neighboring instances with reversed outcomes.
- Research gap: The authors state that prior work had not proposed a black-box-agnostic local method based on both decision and counterfactual rules.This combination defines LORE’s claimed position relative to the surveyed literature.
3 PROBLEM AND EXPLANATIONS
The paper formalizes black-box outcome explanation as learning an interpretable local model and deriving a human-readable explanation for one instance. Its explanation pairs a decision rule with counterfactual rules that reverse the predicted outcome.
- Basic concepts: A predictor maps feature-valued instances to decisions, while a black box is a queryable predictor whose internals are unknown or uninterpretable.The framework restricts attention to binary decisions and supports continuous or categorical features.
- Problem definition: Black-box outcome explanation seeks a human-interpretable explanation for the decision assigned to a particular instance.The approach learns an interpretable predictor that reproduces the black box’s local behavior, then derives the explanation from it.
- Decision rules: A decision rule has a conjunction of feature conditions as its premise and the predicted class as its consequence.An instance is covered when every split condition in the premise evaluates as true.
- Counterfactuals: Counterfactual rules modify feature conditions and represent changes that reverse the predictor’s decision.Minimal counterfactuals minimize the number of split conditions in the modified rule that the original instance does not satisfy.
- Local explanations: A local explanation is a pair consisting of a decision rule satisfied by the instance and counterfactual rules consistent with the alternative outcome.The formal definition represents the explanation as e = ⟨r, Φ⟩.
4 PROPOSED METHOD
LORE generates a balanced, local neighborhood around an instance with a genetic algorithm, learns an interpretable decision tree, and extracts a decision rule plus counterfactual rules. Its neighborhood search targets nearby instances with both matching and differing black-box outcomes while accommodating mixed feature types.
- 4.1 Neighborhood Generation: LORE builds a decision tree from N genetic-algorithm neighbors and extracts one decision rule plus counterfactual rules.The neighbors are labeled by the black box, and the tree is used as the interpretable local predictor.
- 4.1 Neighborhood Generation: The genetic neighborhood is balanced between instances matching and differing from the explained instance’s black-box decision.The two subsets are generated separately and then combined into Z.
- 4.1 Neighborhood Generation: Fitness functions favor instances close to x, unlike x, and selected either for the same or a different black-box outcome.The distance term promotes similarity, while the equality and outcome indicators distinguish the two neighborhood classes.
- 4.1 Neighborhood Generation: The genetic algorithm evolves neighborhoods through fitness-based selection, crossover, mutation, and repeated generations.Crossover recombines parent feature values, while mutation replaces feature values using an empirical feature distribution.
- 4.1 Neighborhood Generation: LORE measures proximity with a weighted combination of categorical simple matching and normalized Euclidean distance for continuous features.The approach is parametric to the distance function and can use improved heterogeneous distances.
- 4.2 Local Rule-Based Classifier and Explanation Extraction: Counterfactual rules retain alternative decision-tree paths with the fewest feature split changes needed to reverse the decision.The tree structure makes the counterfactual changes implicit in paths leading to different labels.
5 EXPERIMENTS
The experiments evaluate LORE on three real-world tabular datasets with several black-box predictors, using fidelity and explanation-quality measures. The setup compares LORE with baselines and competing methods across neighborhood generation, mimicry, and explanation properties.
- Experimental Scope: The experiments analyze genetic-parameter effects, compare local genetic neighborhoods with global approaches, and assess baselines and state-of-the-art competitors.The implementation uses DEAP for neighborhood generation and YADT for decision-tree induction.
- Experimental Setup: Experiments use adult, compas, and german datasets containing both categorical and continuous features.The datasets represent individual people and cover income, risk, and creditor classifications.
- Experimental Setup: The evaluated black boxes are RBF-kernel SVMs, 100-tree random forests, and multi-layer neural networks.Implementations come from scikit-learn, with default parameters unless otherwise stated.
- Experimental Setup: Each dataset is randomly split into 80% training and 20% test instances, with explanations generated for the test set.The training portion fits the black boxes, while the test instances form X for explanation.
- Evaluation Measures: The evaluation compares global fidelity, local-rule fidelity, and counterfactual-rule fidelity between LORE’s tree and the black box.These measures compare predictions on the generated neighborhood or subsets covered by the extracted rules.
- Evaluation Measures: The first three fidelity properties are measured with F1, while hit and c-hit values are averaged over test instances.Aggregated F1 and hit/c-hit scores are reported across the explained test set.
5.2 Analysis of Neighborhood Generation
The neighborhood-generation analysis studies how genetic-algorithm generations and population size affect generation performance, explanation complexity, and runtime on the german dataset.
- Parameter Analysis: The analysis varies generations G and neighborhood size N, reporting fitness, classifier and rule sizes, and elapsed running times.Crossover and mutation probabilities remain fixed at 0.5 and 0.2.
- Parameter Analysis: Results are reported only for german because similar effects were obtained on the other datasets.The analysis covers decision-tree depth, decision-rule antecedent size, and counterfactual complexity.
5.3 Comparing Distance Functions
The distance function used in LORE’s genetic neighborhood generation does not materially change the reported measures on the german dataset. The local approach is also compared with a global predictor, which performs worse on most explanation-quality measures.
- Distance-function comparison: LORE compares normalized Euclidean, cosine, and min-max distances for neighborhood generation on the german dataset.The comparison addresses whether the genetic algorithm’s results depend on the adopted distance function.
- Distance-function comparison: The distance-function comparison does not highlight any considerable difference in the reported measures.The decision-tree-building phase abstracts generated instances into patterns, which may make the approach resilient to distance variability.
- Local versus global: LORE and the global approach achieve similar high hit performance, but LORE considerably overtakes the global approach on fidelity, fairness, and tree depth.The global predictor may produce larger and deeper trees, making its decision rules and counterfactuals more complex to understand.
Local vs Global Explanations.
The paper concludes that extracting an interpretable predictor from each instance’s neighborhood is better than fitting one predictor globally across the test set.
- Local versus global: LORE’s local strategy extracts a predictor from the neighborhood of the instance being explained.
- Local versus global: The global alternative builds a single predictor from all instances in the test set.
- Local versus global: The reported conclusion is that the local approach is better than the global approach.
Comparing Neighborhood Generations.
LORE’s genetic-programming neighborhood generation outperforms the considered baseline generators and produces strong fidelity with lower explanation complexity. Its results also show low variability for most reported measures.
- Baseline neighborhood generators: The baselines include closest-instance selection, random augmentation, CNN-based selection, and random oversampling.These methods are denoted crn, rnd, ris, and ros, respectively.
- Baseline neighborhood generators: LORE overtakes all other neighborhood generators on the aggregated evaluation measures.The comparison aggregates results over various black boxes and datasets.
- Evaluation results: LORE has the highest mean and median f1-measures and the smallest interquartile ranges for the reported distributions.The f1-measures are used to assess fidelity-related mimicry, while the smaller interquartile ranges indicate lower variability.
- Evaluation results: LORE also has the lowest complexity sizes, although cl-fidelity shows its largest variability among the compared methods.For cl-fidelity, LORE’s median remains higher than the competitors’ 90th percentile.
5.5 Comparison with the State-of-Art
LORE is compared with LIME and Anchor using quantitative and qualitative criteria. The reported evidence favors LORE’s fidelity, robustness, rule structure, and treatment of continuous features and counterfactuals.
- LORE versus LIME: Unlike LIME, LORE automatically selects only the features useful for justifying the black-box decision.LIME requires the user to specify the number of features in the explanation.
- LORE versus LIME: LORE has better local fidelity scores and greater robustness than LIME, whose l-fidelity varies substantially across neighborhoods.The paper relates this difference to LORE’s genetic instance generation.
- LORE versus LIME: LORE expresses explanations as decision rules and counterfactual rules, whereas LIME uses feature weights and gives only a rough indication of changes leading toward another decision.The paper presents the rule-based form as more directly connected to the contextual conditions of the decision.
- LORE versus Anchor: Anchor requires prior discretization of continuous features, while LORE’s decision tree can split continuous features directly.The paper also states that Anchor does not clearly extend to computing counterfactuals.
- LORE
LORE is evaluated against Anchor using precision, coverage, and stability measures, alongside example explanations. Its average precision is comparable to Anchor, while coverage and stability favor LORE.
- Figure 9 presents explanations produced by LORE, LIME, and Anchor.
- LORE’s rule precision is on average high and very similar to Anchor’s, whose construction guarantees at least 95% precision.Precision is the fraction of neighborhood instances correctly classified by a rule.
- LORE shows consistently better coverage than Anchor, where coverage measures the fraction of instances to explain covered by the rule.
- LORE has better stability than Anchor for all datasets and black boxes, measured with Jaccard coefficients over feature sets used in repeated rules.Table 5 reports the mean and standard deviation of the Jaccard coefficient.
6 CONCLUSION
The conclusion presents LORE as a local, black-box-agnostic rule-based explanation approach whose genetic neighborhoods support effective explanations. It also identifies global descriptions, human comprehensibility, broader data types, and bias analysis as future directions.
- LORE: LORE builds a decision-tree predictor on artificial instances similar to the explained instance, generated by a genetic algorithm.
- LORE: The resulting local explanation contains one decision rule and counterfactual rules for the reversed decision.
- LORE: Experimental evaluation demonstrates the effectiveness of the genetic neighborhood procedure and reports that LORE outperforms state-of-the-art proposals.
- Future work: Future work includes composing local explanations into a compact global description of the black box.